agentcompass run

agentcompass run creates and executes one evaluation request with the selected benchmark, harness, model, and environment. BENCHMARK HARNESS MODEL are positional arguments in a fixed order; select the environment with --env.

agentcompass run [OPTIONS] BENCHMARK HARNESS MODEL

Run a Minimal Evaluation

The example uses sample_ids in --benchmark-params to select one benchmark task by its stable task ID so you can quickly verify the component and endpoint configuration:

export MODEL_NAME=""
export MODEL_BASE_URL=""
export MODEL_API_KEY=""

agentcompass run \
  swebench_verified \
  mini_swe_agent \
  "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY"

One run command corresponds to one evaluation request. To coordinate multiple explicit requests, use agentcompass launch.

Parameter Reference

This section explains how to use agentcompass run parameters and lists their complete signatures, defaults, and owners. See Run Controls for guidance on concurrency, timeouts, retries, output, and debugging. “Built-in default” means the value used before user-level, project-level, or explicit configuration files override it. A conditional parameter is required only when the selected component or endpoint needs it.

Component Selection and Parameters

Parameter	Required?	Built-in default	What it controls
BENCHMARK	Required	None	Registered benchmark ID. Determines dataset loading, task preparation, verification, and metrics.
HARNESS	Required	None	Registered harness ID. Determines the agent loop or framework used to attempt each task.
MODEL	Required	None	Primary model ID and the model-name segment used in the result path. Prefer "$MODEL_NAME" in shell commands.
--benchmark-params <json>	Conditional	Selected benchmark defaults	Overrides the shared and benchmark-specific fields defined by the selected benchmark config.
--harness-params <json>	Conditional	Selected harness defaults	Overrides the complete parameter schema defined by the selected harness.
--model-base-url <url>	Conditional	""	Base URL of the model API that receives inference requests.
--model-api-key <key>	Conditional	""	Credential used to authenticate with the model API.
--model-api-protocol <protocol>	Optional	Unspecified	Sets the model API protocol. See the linked page for accepted values and selection rules.
--model-params <json>	Optional	{}	Passes fields supported by the selected harness, API protocol, endpoint, and model deployment.
--env <id>	Optional	host_process	Selects where task commands execute. Common choices are docker, daytona, and modal.
--env-params <json>	Conditional	Selected environment and recipe defaults	Overrides the common Environment schema: setup, resources, network policies, and task environment variables.

Configuration and Recipes

Parameter	Required?	Built-in default	What it controls
--config <path>	Optional	None	Loads an additional YAML or JSON override file. Repeat to layer multiple files in command order.
--recipe <id>	Optional	All compatible built-ins	Restricts automatic recipe matching to the specified recipe IDs. Repeat to allow more than one.
--recipe-dir <path>	Optional	[]	Loads a trusted external recipe package directory. Repeat for multiple packages.

Execution Controls

Parameter	Required?	Built-in default	What it controls
--execution-params <json>	Optional	YAML execution defaults	Deep-merges common execution fields, including finalization grace, artifact preparation timeout, and artifact limits. Dedicated CLI options win.
--task-concurrency <int>	Optional	32	Limits concurrent physical attempt executions, including retries.
--max-retries <int>	Optional	0	Retries matching task or scoring failures up to this many times within the current logical attempt.
--retry-pattern-list <json>	Optional	null	Match ERROR message/code; null or [] retries only FATAL. WARNING never retries.
--keep-environment	Optional	Disabled	Skips environment cleanup so task and verifier sandboxes remain available for debugging.

Configure Repeated Attempts

Parameter	Required?	Built-in default	What it controls
--k <int>	Optional	1	Sets the maximum independent attempts per task.
--attempt-strategy <strategy>	Optional	avg	Selects how repeated attempts execute and aggregate.

For example, the following options plan three independent attempts per task and aggregate them with avg:

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --k 3 \
  --attempt-strategy avg

When these options are omitted, AgentCompass uses k=1 and avg: each task runs once and its metrics use native@1. With k>1, attempts are independent execution units and share the --task-concurrency limit; the two strategies differ as follows:

strategy	Compatible Benchmark primary	Execution and result
avg	Scalar score or binary correct	Plans all k attempts and computes avg@k; a binary metric also produces pass@k.
pass	Binary correct only	Stops after the first success or after at most k attempts; produces only the primary metric’s pass@k.

Keep these boundaries in mind:

• A scalar primary with pass raises an error before tasks start.
• k counts logical attempts and excludes retries created within an attempt by --max-retries.
• See Metrics and Aggregation for exact avg@k and pass@k definitions and missing-attempt behavior.

With launch, each request can set its own k and strategy; see Configure k and Strategy per Request.

Output and Reuse

Parameter	Required?	Built-in default	What it controls
--run-name <name>	Optional	""	Adds an optional namespace between results_dir and the combined Model/Benchmark/Harness directory.
--run-id <id>	Optional	Current timestamp	Sets the final run-directory name instead of generating YYYYMMDD_HHMMSS.
--reuse [run-id]	Optional	Disabled	Reuses compatible complete details without errors, or checkpoints for unfinished tasks without a complete detail, from the latest run under the same Model/Benchmark/Harness output namespace or a specified run ID.
--no-checkpoint-resume	Flag	Disabled	Do not use post-agent checkpoints to resume pending fresh evaluations. Complete results and terminal-attempt scheduling records keep their ordinary reuse behavior.

Process Settings

Parameter	Required?	Built-in default	What it controls
--results-dir <path>	Optional	results	Sets the root directory for run outputs.
--data-dir <path>	Optional	data	Sets the root directory for downloaded datasets, caches, and prepared benchmark data.
--timeout-seconds <int>	Optional	360000	Sets the overall timeout in seconds for the evaluation execution phase after component preflight. Explicitly set 0 to disable this outer limit. Component-specific command and verifier timeouts remain separate.
--env-open-qps <provider=qps>	Optional	Local: 0; remote: 10	Limits environment creation rate per provider. Repeat for multiple providers; 0 disables pacing.
--provider-limit <provider=count>	Optional	128 per built-in provider	Sets a process-wide provider limit on physical attempt executions, including retries. Repeat per provider; 0 disables the limit.
--progress <mode>	Optional	auto	Selects terminal progress output: auto, plain, or none.
--log-level <level>	Optional	INFO	Sets console logging to DEBUG, INFO, WARNING, ERROR, or CRITICAL.
--file-log-level <level>	Optional	DEBUG	Sets the run log-file level independently from console logging.
--auto-install-dependencies	Optional	Disabled	Allows AgentCompass to install missing trusted component extras in the active Python environment.

Analysis

Parameter	Required?	Built-in default	What it controls
--enable-analysis	Optional	Enabled	Enables registered post-evaluation analyzers. Use --no-enable-analysis to skip them.
--analysis-params <json>	Optional	Built-in analyzer defaults	Selects analyzers and overrides their concurrency, model, rendering, or analyzer-specific settings.

Component-Specific JSON Parameters

The four JSON parameter flags do not share one schema. Their available fields and defaults depend on the selected component:

Flag	Owner	Where to find its fields
--model-params	Harness, API protocol, and endpoint	Model parameter schema and the endpoint provider’s request schema
--benchmark-params	Selected benchmark	Benchmark parameter schema and agentcompass config docs benchmark <id>
--harness-params	Selected harness	Harness parameter schema and agentcompass config docs harness <id>
--env-params	Selected environment	Environment parameter schema and agentcompass config docs env <id>

sample_ids belongs in --benchmark-params; repeated attempts use the execution controls above. Provider CPU, memory, image, and network settings belong in --env-params. See Configure an Evaluation for the ownership map.