Timeouts

Set a budget for the complete evaluation, each agent execution, or each scoring phase. Start with these three controls; the other settings on this page are for specific collection or timeout problems.

What you want to limit	Setting
All tasks in one run, or all requests in one launch	--timeout-seconds
Agent execution for each attempt	run_timeout_seconds
Evaluation execution for each attempt	evaluation_timeout_seconds

The last two fields belong under execution. The examples below show how to pass them through each interface.

Set Execution and Evaluation Budgets

This example gives the agent 3,600 seconds and scoring 600 seconds, with a 7,200-second limit for the complete evaluation. It uses the same task, Model environment variables, and Docker setup as the Quick Start.

CLI

agentcompass run swebench_verified mini_swe_agent "$MODEL_NAME" \
  --env docker \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --task-concurrency 1 \
  --timeout-seconds 7200 \
  --execution-params '{
    "run_timeout_seconds": 3600,
    "evaluation_timeout_seconds": 600
  }'

YAML

Save the following as timeouts.yaml. Add --config timeouts.yaml to your run command instead of the two timeout options in the CLI example.

runtime:
  timeout_seconds: 7200

execution:
  run_timeout_seconds: 3600
  evaluation_timeout_seconds: 600

Configuration files store reusable settings; the command still selects the Model, Benchmark, Harness, and Environment. See configuration files for the full structure.

Python SDK

import os

from agentcompass import run_evaluation

result = run_evaluation(
    model=os.environ["MODEL_NAME"],
    benchmark="swebench_verified",
    harness="mini_swe_agent",
    environment="docker",
    model_base_url=os.environ["MODEL_BASE_URL"],
    model_api_key=os.environ["MODEL_API_KEY"],
    model_api_protocol="openai-chat",
    benchmark_params={"sample_ids": ["astropy__astropy-12907"]},
    task_concurrency=1,
    timeout_seconds=7200,
    execution_params={
        "run_timeout_seconds": 3600,
        "evaluation_timeout_seconds": 600,
    },
)

For launch, set the overall limit with --timeout-seconds. Put shared task budgets under defaults.execution, or a request’s overrides under requests[].execution.

The overall limit covers task loading, preparation, execution, scoring, analysis, and summarization after component preflight. Its default is 360000 seconds (100 hours); explicitly set 0 to disable it. On expiry, unfinished work is cancelled and resource cleanup begins. Cleanup can continue beyond the deadline.

Task budgets do not add to the overall limit. Allow enough total time for scheduling, all tasks and retries, result collection, and evaluation. The overall limit can expire during any of these phases.

Understand the Defaults

All time values are in seconds. Execution settings accept finite positive values; 0 does not disable a task timeout.

Field	Default	When you would change it
run_timeout_seconds	null: inherit	Give every task an explicit agent execution budget.
evaluation_timeout_seconds	null: inherit	Allow more or less time for grading each task.

Omitting a field or setting it to null inherits the Benchmark plan’s budget, then the task’s budget. Execution can also fall back to the Harness default. If none of these sources supplies a budget, that phase has no time limit. Setting null does not clear an inherited budget.

Each attempt and retry starts a new phase budget. Execution and scoring use separate clocks; unused execution time does not carry over to scoring.

Use a multiplier when tasks already have suitable relative budgets and you want to extend them together. For example, this doubles agent execution time while leaving scoring unchanged:

execution:
  run_timeout_multiplier: 2

Field	Default	Effect
timeout_multiplier	1.0	Scales both execution and evaluation budgets.
run_timeout_multiplier	null	Replaces the common multiplier for execution.
evaluation_timeout_multiplier	null	Replaces the common multiplier for evaluation.

The resolved budget is base seconds × effective multiplier. Phase-specific multipliers replace the common multiplier; they do not multiply together. They also apply to explicit budgets. If there is no base budget, a multiplier cannot create one. Collection budgets and execution compensation are not scaled.

See Which Phase a Timeout Covers

The diagram shows one attempt with a Harness and a separate evaluation Environment. Read it from left to right; column widths do not represent duration. Some Benchmarks skip artifact transfer or use the execution Environment for scoring.

The execution phase includes the Harness’s agent preparation, model requests, and tool loop. Result collection starts afterward with its own budget. The evaluation budget starts after its Environment is ready and artifacts have been restored.

Gray columns are outside the six phase settings shown. Environment creation has a separate startup limit; Harness closing and resource cleanup are not covered by run_timeout_seconds.

Request and Command Limits

run_timeout_seconds and evaluation_timeout_seconds bound an entire phase. Individual model requests, tool commands, or judge requests within that phase may have their own limits. Increasing the phase budget does not increase those limits. The component determines whether an operation timeout triggers a retry, allows execution to continue, or ends execution.

These examples link to the relevant component’s parameter documentation. Fields and defaults depend on the selected component; not every Harness or Benchmark accepts these fields.

Parameter	CLI JSON option	Component and scope
timeout	--model-params	One model request in mini-SWE-agent or OpenHands.
command_timeout	--harness-params	One tool command in mini-SWE-agent or OpenHands.
llm_request_timeout_seconds	--harness-params	One model request in ResearchHarness.
judge_timeout_seconds	--benchmark-params	One judge request in PinchBench, independent of the overall evaluation budget.

For example, this mini_swe_agent command gives the entire agent execution 3,600 seconds, each model request 120 seconds, and each tool command 300 seconds:

agentcompass run swebench_verified mini_swe_agent "$MODEL_NAME" \
  --env docker \
  --model-base-url "$MODEL_BASE_URL" \
  --model-api-key "$MODEL_API_KEY" \
  --model-api-protocol openai-chat \
  --benchmark-params '{"sample_ids":["astropy__astropy-12907"]}' \
  --model-params '{"timeout":120}' \
  --harness-params '{"command_timeout":300}' \
  --execution-params '{"run_timeout_seconds":3600}'

The SDK uses the corresponding model_params, harness_params, and benchmark_params. With --config, put Harness and Benchmark fields directly under their component ID in harnesses or benchmarks. Supply Model parameters through the run command, orchestration request, or SDK. See configuration file structure for the complete format.

Query the selected Harness or Benchmark’s complete field list and defaults with:

agentcompass config docs harness <harness-id>
agentcompass config docs benchmark <benchmark-id>

Advanced Settings

Keep the defaults unless a failure points to one of these operations. Set these fields under execution, through CLI --execution-params, YAML, or SDK execution_params.

Field	Default	Scope
run_timer_compensation_seconds	120	Extra time around Harness execution for preparation and stopping.
runresult_collect_timeout_seconds	60	Read the Harness’s outputs and assemble its answer, trajectory, and diagnostics into RunResult.
artifact_collect_timeout_seconds	null	Total time for the ordered artifact preparation commands; null means no group deadline.
artifact_collect[].timeout_seconds	60	Each artifact preparation command.
artifact_limits.timeout_seconds	600	Each complete batch download or restore.

Compensation, result collection, individual preparation commands, and transfer budgets require finite positive values. See Save and Prepare Artifacts for examples of preparation and transfer limits.

Allow a Harness to Finish Handling Its Timeout

Some Harnesses start their internal execution timer after preparing the agent. The runtime allows an additional 120 seconds by default around Harness execution so that preparation and stopping do not immediately cut off the internal timeout handling. For a resolved execution budget of T seconds, the Harness receives T seconds and the runtime’s outer execution budget is T + 120 seconds.

Increase compensation if agent preparation is unusually slow or logs show the runtime timing out before the Harness finishes stopping. For example:

execution:
  run_timeout_seconds: 5400
  run_timer_compensation_seconds: 300

With this configuration, the agent’s budget remains at 5,400 seconds and the runtime’s outer execution budget is 5,700 seconds. Preparation consumes part of that extra time. Result collection still has its own 60-second budget afterward; it is not included in compensation.

Without a Harness, execution uses the resolved budget directly. Compensation does not create a deadline when execution is unlimited.

Allow More Time to Collect the Result

If execution ends but collecting the answer, logs, or trajectory exceeds 60 seconds, increase the result collection budget:

execution:
  runresult_collect_timeout_seconds: 120

Collection is attempted after execution succeeds, fails, or times out. This budget does not extend agent execution, artifact transfers, or the overall evaluation limit. A partial result preserves available evidence; it does not remove the timeout failure or guarantee a score.

Troubleshoot a Timeout

Inspect the task’s issues and run.log to identify the failing operation before changing a setting. See Diagnose a Failed Run for the files to check.

Where the timeout occurs	What to check
Entire evaluation or orchestration	The overall --timeout-seconds limit.
Creating an Environment	Environment setup.build_timeout_seconds and the provider configuration.
Agent execution	run_timeout_seconds; if the Harness is still preparing or stopping, check compensation.
One tool command or model request	Check Request and Command Limits, such as the selected component’s timeout or command_timeout.
Collecting the answer or trajectory	runresult_collect_timeout_seconds.
Preparing, downloading, or restoring files	The artifact preparation and transfer limits.
Evaluation execution	Check evaluation_timeout_seconds for the whole phase, or the Benchmark’s request limit, such as PinchBench’s judge_timeout_seconds, for one judge request.