Implementation Overview

Before implementing a Benchmark, decide who runs the agent, where evaluation occurs, and how results are aggregated.

There is no single Benchmark template for every use case. Most integrations delegate execution to a Harness; a Benchmark takes over execution only when the upstream interaction loop is part of the evaluation definition. Scoring can run in the AgentCompass process, the task Environment, or a separate evaluation Environment.

Step 1: Choose the Execution Owner

This choice determines whether the Harness or the Benchmark runs the agent loop:

Scenario	Implementation type	Who implements the execution-stage run_task()	Harness in the command
An existing Harness can run the prepared task	BaseBenchmark	Harness	A registered Harness ID
The upstream Benchmark defines a dedicated interaction loop that cannot be reused	HarnessFreeBenchmark	Benchmark	none

HarnessFreeBenchmark is a subclass of BaseBenchmark. They share the task-loading, preparation, artifact-collection, evaluation, and aggregation contracts; only the owner of the execution stage differs. Prefer a Harness-driven Benchmark. Use a Benchmark-driven implementation only when the upstream interaction protocol itself belongs to the Benchmark definition.

Step 2: Choose the Evaluation Location

This choice determines where evaluate() runs and is independent of the execution owner:

Evaluator requirement	evaluation_environment_mode	env passed to evaluate()
Score only an answer or already collected data	none	None
Inspect the same workspace or process state left by task execution	reuse	The still-open task Environment
Run the verifier in an isolated Environment	fresh	A new evaluation Environment

Both BaseBenchmark and HarnessFreeBenchmark can use all three modes. Use TaskSpec.artifacts for shared collection and fresh restore: generic tasks have no implicit paths, while the Harbor adapter adds /logs/artifacts/ even for empty declarations. Execution parameters may replace those path and command lists; use execution.save_artifacts to control local storage. Declare TaskSpec.artifact_collect commands when files need preparation; otherwise runtime collects existing files directly. See Evaluation Modes and Artifacts.

Step 3: Choose the Aggregation Strategy

This choice determines how attempt-level verdicts become request-level metrics. It does not change the execution owner or evaluation location:

Result shape	aggregate_metrics()
The official result is a mean of the Contract’s reduced task observations	Inherit the default aggregation from BaseBenchmark
The official result uses a ratio of sums, total, rank, medal, or another corpus formula	Override the method and return MetricReport

See Results and Aggregation for the result fields and aggregation patterns. Regardless of the final combination, read Shared Contracts first to define task fields, visibility boundaries, and per-task plans.

Shared Lifecycle

Each request loads and selects tasks first, then performs these steps for every task and attempt:

load_tasks
  → select_tasks
  → build_plan
  → open the task Environment
  → prepare_task
  → Harness.run_task or Benchmark.run_task
  → artifact_collect commands (when declared and enabled)
  → download_artifacts (runtime)
  → evaluate
  → persist the attempt result
  → aggregate_metrics

The task Environment remains running during artifact preparation and collection. The evaluation mode determines where evaluate() runs. After all tasks finish, aggregate_metrics() reads persisted results and produces the summary.

Method Responsibilities

Method	Requirement or default	Responsibility
load_tasks()	Required	Convert a pinned dataset release into TaskSpec values
select_tasks()	Default implementation	Apply the shared task-selection logic; override only for special selection semantics
build_plan()	Default implementation	Resolve typed Benchmark state for one task attempt
prepare_task()	Required	Prepare execution input for the Harness or Benchmark inside the task Environment
evaluate()	Required	Preserve execution status and write declared observations to RunResult.metrics
aggregate_metrics()	Default Contract-driven aggregation	Convert reduced task observations into a MetricReport
run_task()	Required only for HarnessFreeBenchmark	Run the inference or interaction loop owned by the Benchmark

After the implementation works, add the user-facing entry point through Documentation Update, then follow Validation and Alignment with real data, Environments, and official results.

Declare collect_artifacts = False on the Benchmark class to mark shared artifact collection as unsupported. Runtime rejects conflicting task/Recipe declarations and user path/command overrides before environment allocation. The default is True; having no default artifact paths alone does not disable the capability.