Choose a Benchmark

Select a registered benchmark and configure its complete benchmark-parameter schema.

Benchmarks define what is evaluated. Each benchmark owns its dataset, stable task ids, task preparation, scoring logic, and aggregate metrics. Select the benchmark as the first positional argument to agentcompass run:

agentcompass run <benchmark> <harness> "$MODEL_NAME"

Find a Benchmark

Use the live registry to see the benchmarks available in your installed AgentCompass revision:

agentcompass list benchmark

The sidebar links to benchmarks with dedicated task, parameter, compatibility, and run documentation.

Configure Benchmark Parameters

The Run Parameter Reference introduces --benchmark-params <json>. The <json> value is one JSON object containing the complete parameter override for the selected benchmark:

agentcompass run \
  <benchmark> \
  <harness> \
  "$MODEL_NAME" \
  --benchmark-params '{
    "sample_ids": ["<task-id>"],
    "<benchmark-specific-field>": "<value>"
  }'

The accepted object combines two schemas:

benchmark params
  ├─ shared fields from RuntimeBenchmarkConfig
  └─ fields defined by the selected benchmark config

Shared Benchmark Fields

Every benchmark config derived from RuntimeBenchmarkConfig supports these user-facing fields. The table shows base defaults; the selected Benchmark can override them.

Field	Type	Base default	Meaning and when to change it
sample_ids	list[str] | null	null	Runs only the listed stable task ids. Use it for a smoke test, failed-task rerun, or a controlled subset. Unknown ids fail before execution.
aggregation_mode	”micro_weighted” | “category_mean"	"micro_weighted”	Selects how generic metrics combine tasks and categories when category_hierarchy is not set.
category_hierarchy	object | null	null	Uses an explicit category aggregation tree and takes precedence over aggregation_mode. Leave unset unless the Benchmark documentation defines one.

Repeated attempts are configured under execution.attempts, not in this object. See Metrics and Aggregation for the attempt plan, Metric Contracts, and how the aggregation fields above combine task results.

The model id is not part of this JSON object. It remains the third positional argument to agentcompass run and is injected into the benchmark config by the runtime.

Each benchmark also adds its own fields to the shared schema. Whether or not it has a dedicated page, query the complete field list, types, defaults, and descriptions directly from the installed code:

agentcompass config docs benchmark <benchmark-id>

When a dedicated benchmark page exists, use it as the source for valid values, recommended settings, required credentials, and field interactions.

Build the JSON Object

For example, swebench_verified combines shared task-selection fields with its own preparation fields:

{
  "sample_ids": ["astropy__astropy-12907"],
  "prepare_mode": "prebaked",
  "workspace_root": "/testbed"
}

This expanded object demonstrates ownership; it is not a recommendation to repeat defaults in every command. Pass only the fields that must differ from the selected benchmark’s effective configuration.

Set the entire evaluation phase budget with evaluation_timeout_seconds under execution. The selected Benchmark defines limits for individual operations such as judge requests, passed through --benchmark-params, for example PinchBench’s judge_timeout_seconds. See Request and Command Limits for configuration locations and component parameters.

--benchmark-params must be valid JSON, so keys and string values use double quotes. CLI values override matching fields under the Benchmark ID in the configuration file’s benchmarks section. Inspect the merged built-in and configuration-file values before adding the final CLI override:

agentcompass config show \
  --benchmark <benchmark-id> \
  --config <config-file>

Images and Provider Settings

Heavyweight benchmarks usually attach task images, workspace roots, and resource hints to task metadata. Compatible recipes translate those requirements for Docker, Daytona, or Modal. Keep provider image, resource, and network overrides in --env-params; they are not benchmark parameters.