> ## Documentation Index
> Fetch the complete documentation index at: https://opencompass-docs-preview-pr-335-0.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Implementation Overview

Before implementing a Benchmark, decide who runs the agent, where evaluation occurs, and how results are aggregated.

There is no single Benchmark template for every use case. Most integrations delegate execution to a Harness; a Benchmark takes over execution only when the upstream interaction loop is part of the evaluation definition. Scoring can run in the AgentCompass process, the task Environment, or a separate evaluation Environment.

## Step 1: Choose the Execution Owner

This choice determines whether the Harness or the Benchmark runs the agent loop:

| Scenario | <span style={{ display: "inline-block", minWidth: "10.5rem" }}>Implementation type</span> | Who implements the execution-stage `run_task()` | Harness in the command |
| - | - | - | - |
| An existing Harness can run the prepared task | `BaseBenchmark` | Harness | A registered Harness ID |
| The upstream Benchmark defines a dedicated interaction loop that cannot be reused | `HarnessFreeBenchmark` | Benchmark | `none` |

`HarnessFreeBenchmark` is a subclass of `BaseBenchmark`. They share the task-loading, preparation, artifact-collection, evaluation, and aggregation contracts; only the owner of the execution stage differs. Prefer a [Harness-driven Benchmark](/en/developer_guide/extensions/benchmark/code_implementation/harness_driven). Use a [Benchmark-driven implementation](/en/developer_guide/extensions/benchmark/code_implementation/benchmark_driven) only when the upstream interaction protocol itself belongs to the Benchmark definition.

## Step 2: Choose the Evaluation Location

This choice determines where `evaluate()` runs and is independent of the execution owner:

| Evaluator requirement | `evaluation_environment_mode` | `env` passed to `evaluate()` |
| - | - | - |
| Score only an answer or already collected data | `none` | `None` |
| Inspect the same workspace or process state left by task execution | `reuse` | The still-open task Environment |
| Run the verifier in an isolated Environment | `fresh` | A new evaluation Environment |

Both `BaseBenchmark` and `HarnessFreeBenchmark` can use all three modes. Use `TaskSpec.artifacts` for shared collection and fresh restore: generic tasks have no implicit paths, while the Harbor adapter adds `/logs/artifacts/` even for empty declarations. Execution parameters may replace those path and command lists; use `execution.save_artifacts` to control local storage. Declare `TaskSpec.artifact_collect` commands when files need preparation; otherwise runtime collects existing files directly. See [Evaluation Modes and Artifacts](/en/developer_guide/extensions/benchmark/code_implementation/evaluation_modes).

## Step 3: Choose the Aggregation Strategy

This choice determines how attempt-level verdicts become request-level metrics. It does not change the execution owner or evaluation location:

| Result shape | `aggregate_metrics()` |
| - | - |
| The official result is a mean of the Contract's reduced task observations | Inherit the default aggregation from `BaseBenchmark` |
| The official result uses a ratio of sums, total, rank, medal, or another corpus formula | Override the method and return `MetricReport` |

See [Results and Aggregation](/en/developer_guide/extensions/benchmark/code_implementation/results_and_aggregation) for the result fields and aggregation patterns. Regardless of the final combination, read [Shared Contracts](/en/developer_guide/extensions/benchmark/code_implementation/shared_contracts) first to define task fields, visibility boundaries, and per-task plans.

## Shared Lifecycle

Each request loads and selects tasks first, then performs these steps for every task and attempt:

```text theme={"system"}
load_tasks
  → select_tasks
  → build_plan
  → open the task Environment
  → prepare_task
  → Harness.run_task or Benchmark.run_task
  → artifact_collect commands (when declared and enabled)
  → download_artifacts (runtime)
  → evaluate
  → persist the attempt result
  → aggregate_metrics
```

The task Environment remains running during artifact preparation and collection. The evaluation mode determines where `evaluate()` runs. After all tasks finish, `aggregate_metrics()` reads persisted results and produces the summary.

## Method Responsibilities

| <span style={{ display: "inline-block", minWidth: "10.5rem" }}>Method</span> | Requirement or default | Responsibility |
| - | - | - |
| `load_tasks()` | Required | Convert a pinned dataset release into `TaskSpec` values |
| `select_tasks()` | Default implementation | Apply the shared task-selection logic; override only for special selection semantics |
| `build_plan()` | Default implementation | Resolve typed Benchmark state for one task attempt |
| `prepare_task()` | Required | Prepare execution input for the Harness or Benchmark inside the task Environment |
| `evaluate()` | Required | Preserve execution status and write declared observations to `RunResult.metrics` |
| `aggregate_metrics()` | Default Contract-driven aggregation | Convert reduced task observations into a `MetricReport` |
| `run_task()` | Required only for `HarnessFreeBenchmark` | Run the inference or interaction loop owned by the Benchmark |

After the implementation works, add the user-facing entry point through [Documentation Update](/en/developer_guide/extensions/benchmark/documentation_update), then follow [Validation and Alignment](/en/developer_guide/extensions/benchmark/validation_and_alignment) with real data, Environments, and official results.

Declare `collect_artifacts = False` on the Benchmark class to mark shared artifact collection as unsupported. Runtime rejects conflicting task/Recipe declarations and user path/command overrides before environment allocation. The default is `True`; having no default artifact paths alone does not disable the capability.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.