> ## Documentation Index
> Fetch the complete documentation index at: https://opencompass-docs-preview-pr-335-0.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Python SDK

The Python SDK and CLI use the same evaluation runtime. Choose an entry point based on the number of evaluation
requests:

| Request type | Synchronous entry point | Asynchronous entry point | CLI equivalent |
| - | - | - | - |
| Single evaluation request | `run_evaluation()` | `async_run_evaluation()` | [`agentcompass run`](/en/user_guide/using_agentcompass/cli/run) |
| Multiple evaluation requests | `launch()` | `async_launch()` | [`agentcompass launch`](/en/user_guide/using_agentcompass/cli/launch) |

## Single Evaluation Request

`run_evaluation()` executes one request composed of a [Model](/en/user_guide/modules/models/overview),
[Benchmark](/en/user_guide/modules/benchmarks/overview), [Harness](/en/user_guide/modules/harnesses/overview), and
[Environment](/en/user_guide/modules/environments/overview):

```python theme={"system"}
import os

from agentcompass import run_evaluation

result = run_evaluation(
    benchmark="swebench_verified",
    harness="mini_swe_agent",
    model=os.environ["MODEL_NAME"],
    environment="docker",
    benchmark_params={"sample_ids": ["astropy__astropy-12907"]},
    model_base_url=os.environ["MODEL_BASE_URL"],
    model_api_key=os.environ["MODEL_API_KEY"],
    model_api_protocol="openai-chat",
    model_params={"temperature": 0},
    task_concurrency=1,
    results_dir="results",
    progress="auto",
)
```

All arguments are keyword-only. On success, the function returns a dictionary containing `metadata`, `metrics`,
`summary`, and `paths`; per-task details are persisted in the result directory. A timeout or execution failure raises
the corresponding exception.

In an asynchronous application, use `await async_run_evaluation(...)`. It accepts the same arguments and returns the
same value as the synchronous entry point.

## Multiple Evaluation Requests

`launch()` accepts an `OrchestrationSpec`. Each `RunRequestSpec` represents one named evaluation request, while
`OrchestrationDefaults` stores components and settings shared by every request:

```python theme={"system"}
import os

from agentcompass import (
    OrchestrationDefaults,
    OrchestrationSpec,
    RunRequestSpec,
    launch,
)

spec = OrchestrationSpec(
    name="terminal-evaluations",
    task_concurrency=4,
    defaults=OrchestrationDefaults(
        harness={"id": "terminus2", "max_turns": 300},
        environment={"id": "docker"},
        model={
            "id": os.environ["MODEL_NAME"],
            "base_url": os.environ["MODEL_BASE_URL"],
            "api_key": os.environ["MODEL_API_KEY"],
            "api_protocol": "openai-chat",
        },
    ),
    requests=[
        RunRequestSpec(
            name="terminal-bench-2-verified",
            benchmark={"id": "terminal_bench_2_verified"},
        ),
        RunRequestSpec(
            name="terminal-bench-2.1",
            benchmark={"id": "terminal_bench_2_1"},
        ),
    ],
)

result = launch(spec, progress="auto")
```

`task_concurrency` is the physical attempt-execution limit shared by all requests; retries consume the same pool. `launch()` returns an
`OrchestrationResult`: `status` records the orchestration status, and `requests` stores each named request's status,
result, error, and output paths. A failure in one request does not discard results from other requests.

Multi-request parameters are divided among orchestration-wide settings, defaults shared by every request, and
per-request overrides. Put them in `OrchestrationSpec`, `OrchestrationDefaults`, and the corresponding
`RunRequestSpec`, respectively.

In an asynchronous application, use `await async_launch(spec, ...)`. See
[`agentcompass launch`](/en/user_guide/using_agentcompass/cli/launch#mapping-rules) for orchestration inheritance and
mapping rules.

## CLI Parameter Mapping

The CLI receives command-line strings, while the SDK uses `snake_case` keywords and native Python objects. The tables
below first list parameters shared by the `run` and `launch` commands and both SDK entry points, followed by the input
forms specific to a single request and a multi-request orchestration.

### Shared Runtime Parameters

| CLI | Python SDK | Representation |
| - | - | - |
| `--config <path>` | `config_path` | Repeatable in the CLI; the SDK accepts one path or a sequence of paths. `launch()` accepts this argument only with an `OrchestrationSpec`. |
| `--task-concurrency <n>` | `task_concurrency` | Limits concurrent physical attempt executions, including retries, within one request or across the whole orchestration. |
| `--results-dir <path>` | `results_dir` | Sets the result root directory. |
| `--data-dir <path>` | `data_dir` | Sets the data and cache root directory. |
| `--timeout-seconds <seconds>` | `timeout_seconds` | Limits one evaluation request or the whole orchestration. Single-request calls accept integer seconds; multi-request calls also accept fractional values. |
| `--provider-limit <provider>=<n>` | `provider_limits` | Repeatable in the CLI; the SDK accepts `dict[str, int]`. |
| `--env-open-qps <provider>=<qps>` | `env_open_qps` | Repeatable in the CLI; the SDK accepts `dict[str, float]`. |
| `--progress auto\|plain\|none` | `progress` | The SDK accepts the same string values. |
| `--log-level <level>` | `log_level` | One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`. |
| `--file-log-level <level>` | `file_log_level` | Accepts the same values as `log_level`. |
| `--auto-install-dependencies` | `auto_install_dependencies` | The SDK accepts a Boolean. |
| None | `log_file` | The SDK can set the log-file path. |
| None | `on_progress` | The SDK can receive progress-event callbacks. |

### Direct Parameters for a Single Evaluation Request

| `agentcompass run` | `run_evaluation()` / `async_run_evaluation()` | Representation |
| - | - | - |
| `BENCHMARK` | `benchmark` | Benchmark ID; keyword-only in the SDK. |
| `HARNESS` | `harness` | Harness ID; keyword-only in the SDK. |
| `MODEL` | `model` | Model ID; keyword-only in the SDK. |
| `--benchmark-params <json>` | `benchmark_params` | The CLI accepts a JSON object; the SDK accepts a `dict`. |
| `--harness-params <json>` | `harness_params` | The CLI accepts a JSON object; the SDK accepts a `dict`. |
| `--model-base-url <url>` | `model_base_url` | The value maps directly. |
| `--model-api-key <key>` | `model_api_key` | The value maps directly. |
| `--model-api-protocol <protocol>` | `model_api_protocol` | The SDK accepts a protocol name, `auto`, or a list of strings directly. |
| `--model-params <json>` | `model_params` | The CLI accepts a JSON object; the SDK accepts a `dict`. |
| `--env <id>` | `environment` | Environment ID. |
| `--env-params <json>` | `environment_params` | The CLI accepts a JSON object; the SDK accepts a `dict`. |
| `--execution-params <json>` | `execution_params` | Common execution overrides; deep-merged over YAML, including artifact budgets. |
| `--max-retries <n>` | `max_retries` | The value maps directly. |
| `--retry-pattern-list <json>` | `retry_pattern_list` | The CLI accepts a JSON string array; the SDK accepts `list[str]`. |
| `--recipe <id>` | `enabled_recipes` | The CLI accepts a repeatable [Recipe](/en/user_guide/other_features/recipes) ID; the SDK accepts a string list. |
| `--recipe-dir <path>` | `recipe_dirs` | Repeatable in the CLI; the SDK accepts a sequence of paths. |
| `--run-name <name>` | `run_name` | The value maps directly. |
| `--run-id <id>` | `run_id` | Sets the ID for a new result directory. |
| `--reuse [run-id]` | `reuse`, `reuse_run_id` | The SDK separates the reuse switch from the ID of the run to reuse. |
| `--no-checkpoint-resume` | `checkpoint_resume=False` | Disable post-agent checkpoint recovery for pending fresh evaluations; the effective default is `True`. |
| `--keep-environment` | `keep_environment` | The SDK accepts a Boolean. |
| `--enable-analysis` | `enable_analysis` | The SDK accepts a Boolean. |
| `--analysis-params <json>` | `analysis_params` | The CLI accepts a JSON object; the SDK accepts a `dict`. |

See the [`agentcompass run` parameter reference](/en/user_guide/using_agentcompass/cli/run#parameter-reference) for
meanings and defaults.

### Orchestration Parameters for Multiple Evaluation Requests

| `agentcompass launch` | `launch()` / `async_launch()` | Representation |
| - | - | - |
| `ORCHESTRATION_PATH` | `orchestration` | The CLI reads a YAML or JSON file; the SDK accepts an `OrchestrationSpec` or resolved `Orchestration` object. |
| `--cleanup-grace-seconds <seconds>` | `cleanup_grace_seconds` | Sets the cooperative cleanup grace period after cancellation. |
| `--run-id <id>` | No same-named keyword | The CLI overrides every request's `output.run_id`; in the SDK, set each `RunRequestSpec.output`. |
| `--reuse` | No same-named keyword | Corresponds to `reuse: true` in `OrchestrationDefaults.runtime`; an individual request can override the default. |
| `--dry-run` | None | Only the CLI provides orchestration preflight and resolved-output display. |
| `runtime.recipe_dirs` in the orchestration file | `OrchestrationSpec.runtime.recipe_dirs` | Multi-request evaluation has no corresponding CLI option or `launch()` keyword argument. |
| None | `on_request_finished` | The SDK can receive a callback whenever a request finishes. |

The top-level fields of `OrchestrationSpec` are `version`, `name`, `task_concurrency`, `runtime`, `defaults`, and
`requests`. The only currently supported `version` is `1`.

### Request Fields in an Orchestration

A single evaluation contains the same components and request settings shown below, but passes them directly to
`agentcompass run` or `run_evaluation()`. In a multi-request orchestration, these values are not `launch()` keyword
arguments: the CLI places them in the orchestration file, while the SDK places them in `OrchestrationDefaults` or
`RunRequestSpec`.

| Orchestration file | Python SDK | Fields |
| - | - | - |
| `requests[].name` | `RunRequestSpec.name` | A required, unique name for each request. |
| `defaults.benchmark` / `requests[].benchmark` | `benchmark` | `id` and same-level Benchmark configuration fields. |
| `defaults.harness` / `requests[].harness` | `harness` | `id` and same-level Harness configuration fields. |
| `defaults.model` / `requests[].model` | `model` | `id`, `base_url`, `api_key`, `api_protocol`, and `params`. |
| `defaults.environment` / `requests[].environment` | `environment` | `id` and same-level Environment configuration fields. |
| `defaults.execution` / `requests[].execution` | `execution` | `max_retries`, `retry_pattern_list`, `enabled_recipes`, `keep_environment`, `enable_analysis`, and `analysis_params`. |
| `defaults.runtime` / `requests[].runtime` | `runtime` | `reuse`, `reuse_run_id`, and `checkpoint_resume`. |
| `defaults.output` / `requests[].output` | `output` | `run_name` and `run_id`. |

`task_concurrency` is orchestration-level only; it cannot appear under `defaults.execution` or
`requests[].execution`. See [`agentcompass launch`](/en/user_guide/using_agentcompass/cli/launch#mapping-rules) for
the complete field structure and inheritance rules.

## Related Pages

* Concurrency, retries, and recovery: [Run Controls](/en/user_guide/using_agentcompass/run_controls)
* Overall and per-task budgets: [Timeouts](/en/user_guide/using_agentcompass/timeouts)
* Saving task files: [Save and Prepare Artifacts](/en/user_guide/using_agentcompass/artifacts)
* Configuration-file loading and merging: [`agentcompass config`](/en/user_guide/using_agentcompass/cli/config)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.