Run Records and Diagnostics

When an evaluation request starts writing results, AgentCompass creates a dedicated run directory for it. In addition to task details and aggregate results, the directory contains these run records:

<run-dir>/
├── run_info.json
├── params.json
├── progress.json
├── progress.jsonl
└── run.log

run_info.json records the request configuration, metric-artifact provenance, and final state. params.json keeps the compact parameter set needed to save and re-aggregate results. progress.json provides the latest snapshot, progress.jsonl preserves the full event sequence, and the log records readable execution messages and errors.

When Files Are Created

File	When it is created and updated
run.log	Created when the run directory is reserved and receives logs from that point onward.
run_info.json	Created before task loading. It is updated for resolved execution plans, generated metric artifacts, and the request’s final state.
progress.json, progress.jsonl	Created with the first progress event. Every later event updates the snapshot and is appended to the event stream.
params.json	Created or rewritten when an evaluation saves task details. It is rewritten after that evaluation’s final aggregation succeeds, even when no tasks were selected.

Not every invocation leaves these files behind. The CLI and SDK check the request before creating its run directory; a failure at that point creates no run directory. agentcompass launch --dry-run also creates no output.

A preparation error after the directory exists will usually leave the log and run_info.json. If error handling completes normally, AgentCompass also writes the final state and a run_finished event. If the process is forcibly terminated, the final state, the last progress events, or params.json may not have been written.

run_info.json

run_info.json records which request configuration the evaluation used, which plan produced the current metric artifacts, and how the request ended. It is created before tasks are loaded and updated throughout the run.

Top-Level Fields

Field	Meaning
schema_version	Exact run-record schema. The current value is agentcompass.run_info.v3.
run_id	The final run ID assigned to this request.
started_at	Time when this record was created, in time-zone-aware ISO 8601 format. It is not the start time of the AgentCompass process or the entire orchestration.
request	Request produced after CLI, configuration-file, or SDK values are merged according to configuration precedence. Per-task Recipes have not yet been applied.
metric_artifacts	Present after summary.md and metrics.json are generated. Identifies their source and exact report plan.
reused_from	Present when a reuse-source run is resolved. It records the source run’s run_id, path, or both, and can appear even when no task is ultimately reused.
resolved_execution_plans	Present after at least one task attempt resolves a plan. It records plan summaries by task ID and attempt number.
status	Final request state: completed, failed, cancelled, or timed_out. It may be absent until the request finishes normally.
finished_at	Time when the final state was written, in time-zone-aware ISO 8601 format.
error	Error information when the request ends with an error; omitted after successful completion.

Current run-info v3 uses structured issues. Previous v2 results are adapted only at read boundaries; classification-unknown failures cannot be reused for current scoring. Historical directories are not modified.

request Structure

request is divided into model, Benchmark, Harness, Environment, execution, runtime, output, and metadata sections. Each component’s params is an open object whose fields depend on the selected component.

Field path	Meaning
model.id	ID of the model being evaluated.
model.base_url	Base address of the model API; it may be empty when not configured.
model.api_key	Model API credential. It is redacted according to sensitive-key rules when written, so the original secret cannot be recovered from this value.
model.api_protocol	Model API protocol name or ordered protocol list. auto and an unspecified value are normalized to an empty string while the request is built, so the literal auto is not persisted here.
model.params	Request or generation parameters passed to the model client.
benchmark.id	Component ID of the selected Benchmark.
benchmark.params	Benchmark-specific parameters after configuration and request overrides are merged.
harness.id	Component ID of the selected Harness.
harness.params	Harness-specific parameters after configuration and request overrides are merged.
environment.id	Component ID of the selected Environment.
environment.params	Environment-specific parameters after configuration and request overrides are merged. Per-task Recipe changes are not included yet.
environment.network_policy	Network policy used while preparing the Environment.
environment.run_network_policy	Optional network policy used by the Harness or task execution. It may be omitted when not configured separately.
environment.verifier_network_policy	Optional network policy used during Benchmark scoring. It may be omitted when not configured separately.
execution.task_concurrency	Maximum concurrent physical attempt executions, including retries.
execution.attempts	Exact repeated-attempt plan containing k and strategy.
execution.enabled_recipes	Recipe IDs eligible for matching. An empty list leaves all candidate Recipes eligible.
execution.keep_environment	Whether to preserve an Environment after the task for debugging.
execution.enable_analysis	Whether to run analyzers during evaluation.
execution.analysis_params	Analyzer selection, analysis model, and analyzer-specific settings.
execution.max_retries	Maximum number of runtime retries within each evaluation attempt.
execution.retry_pattern_list	Regexes matching ERROR message/code. Null and [] retry only FATAL.
runtime.reuse	Reuse compatible results per attempt under the current issues, pattern and retry-budget policy.
runtime.reuse_run_id	Run ID to use as the reuse source. When empty, AgentCompass may find the latest compatible run.
runtime.checkpoint_resume	Whether pending fresh evaluations can resume from a clean post-agent checkpoint. Effective default: true.
output.run_name	Optional namespace below the result root.
output.run_id	Directory ID ultimately used for this run.
metadata.config_path	Configuration files loaded to build the request. One file is stored as a path string; multiple files are stored as a JSON-array string containing every path. Omitted when no configuration file was loaded.
metadata.recipe_dirs	External Recipe directories loaded to build the request; omitted when absent.

Each network-policy object contains network_mode, which selects the network-access mode, and allowed_hosts, which lists permitted hosts. When AgentCompass writes JSON, it removes null values and empty objects or lists. An empty allowed_hosts may therefore be absent from the file.

request is not a copy of the original command line. It also excludes process-level settings such as results_dir, the whole-request timeout, log levels, and Environment provider concurrency limits. To verify these settings, consult the invocation, configuration, and log together. See the Model, Benchmark, Harness, and Environment documentation for component-specific fields.

When present, reused_from has these fields:

Field	Meaning
run_id	Run ID of the reuse source.
path	Path to the source run directory.

Reuse validation

Reuse requires a supported run schema, the same Benchmark ID, and the same execution.attempts plan. Tasks and checkpoints are matched by task ID and attempt number. Category labels, complete task inputs, Model parameters, Harness settings, Environment settings, and Recipe configuration do not need to match the source run.

New records do not write execution_fingerprints or task_fingerprints. These fields in older records are ignored; their absence does not prevent reuse or summary generation. The sanitized request and resolved plans remain available for provenance.

Selecting reuse means using the saved agent outputs and prepared task context. Pending fresh evaluations use the current request and a newly resolved execution plan, including evaluation timeouts, resources, and environment variables. Complete results still take precedence and are not evaluated again when parameters change.

Checkpoint recovery validates task/attempt identity, a completed agent run, fresh evaluation mode, artifact coverage, and file integrity. Saved artifacts may include more declarations than the current verifier needs; declaration order and collection settings do not need to match. A newly required artifact declaration that was never captured cannot be restored. Recorded missing or excluded outputs retain their original meaning.

metric_artifacts Structure

Whenever the Benchmark metric files are written, AgentCompass replaces this provenance record:

{
  "metric_artifacts": {
    "generated_at": "<ISO-8601-time>",
    "source": "evaluation",
    "report": {
      "k": 3,
      "strategy": "avg",
      "aggregation": "micro_weighted"
    }
  }
}

source is evaluation for normal run finalization and summary after a non-dry-run agentcompass summary. A summary regeneration also records a redacted benchmark_params_override object, including an empty object when no override was passed. report binds the files to the exact attempt plan and run-level aggregation that produced them; it does not replace the original request.

resolved_execution_plans Structure

resolved_execution_plans records the Environment, network policies, and Recipes resolved for each task attempt. Its structure is:

{
  "resolved_execution_plans": {
    "<task-id>": {
      "attempts": {
        "1": {
          "environment": {
            "id": "<environment-id>",
            "network_policy": {
              "network_mode": "public",
              "allowed_hosts": []
            }
          },
          "evaluation_environment": null,
          "run_network_policy": {
            "network_mode": "public",
            "allowed_hosts": []
          },
          "verifier_network_policy": {
            "network_mode": "public",
            "allowed_hosts": []
          },
          "applied_recipes": []
        }
      }
    }
  }
}

Field or key	Meaning
<task-id>	Task ID provided by the Benchmark.
attempts	Plan records for the task, keyed by attempt number. Attempt numbers start at 1.
environment	Environment planned for task execution. It records only the component ID and preparation network policy.
evaluation_environment	Separate Environment planned for scoring. It records only the component ID and preparation network policy, and is null when no separate scoring environment is needed.
run_network_policy	Network policy planned for Harness or task execution.
verifier_network_policy	Network policy planned for Benchmark scoring.
applied_recipes	Recipe IDs matched for this attempt.

The plan summary is written after resolution but before the Environment is opened. It tells you what the attempt planned to use; it does not prove that the Environment was created successfully. It also excludes the complete Recipe-resolved image, snapshot, working directory, resources, and Environment provider parameters.

A task or attempt reused without execution receives no new resolved-plan entry. The task detail keeps the metric attempt plan in attempt_plan; Environment and Recipe plans remain in the source run’s run_info.json.

Evaluation Checkpoints

<run-dir>/details/<state>/<readable-task-id>--<sha256>/
├── task.json
└── attempt-<n>/
    ├── result.json
    ├── checkpoint.json
    └── artifacts/

task.json stores shared task metadata, the attempt plan, and logical attempt numbers mapped to directory names such as "1": "attempt-1". Each attempt’s result.json stores its own result and retry count. Readers derive task summaries and total retries; there is no task-level result.json. Evaluation checkpoints record agent completion and references to saved artifacts.

checkpoint.json uses schema agentcompass.attempt_checkpoint.v1. Its evaluation section stores task/attempt identity, agent completion, source run and evaluation mode, and the artifact manifest and references the same artifacts/ tree used by the result. The runtime no longer writes an additional ZIP. Its scheduler section records terminal attempt state until the task result is durable; an in-progress download can also record artifact_transfer diagnostics. Metadata updates are atomic.

Recovery validates task/attempt identity, declarations, file sizes and checksums before uploading artifacts into the fresh verifier. Cross-run reuse preserves the logical attempt number, copies validated artifacts into the target’s attempt-<n> directory and updates references before publishing results. When a task moves between running/ and its state directory, artifact references are updated with it. The target run can then be used independently of its source.

Default artifact limits remain 16 GiB, 100,000 entries and 600 seconds per operation. Files retain their original bytes. When an attempt reruns, previous local outputs are moved beneath that attempt’s retries/ directory, so stale files cannot mix with the new submission. No per-attempt logs directory is created automatically.

Evaluation checkpoint v4 saves a classified RunResult, PreparedTask, network policy and artifact manifest. Each none/fresh scoring retry receives an isolated copy. Legacy v3 records remain readable; missing classified snapshots prevent cross-run materialization and cause normal execution with an explicit reason.

The Benchmark rebuilds evaluation context from the current task and plan through prepare_evaluation(). SWE-bench evaluators load their patch from the saved artifact. The runtime does not require or validate a trajectory to resume evaluation. Benchmarks that need additional in-memory run state must implement artifact-based recovery before they can resume this way; otherwise the agent reruns. Missing required inputs also cause a rerun. Successful complete results remain reusable without reevaluation.

params.json

params.json stores only the parameters needed to write task details and regenerate summaries. AgentCompass rewrites it when saving task details or generating the final summary. The file may not exist if the request fails before either operation. Running agentcompass summary separately leaves an existing params.json unchanged, while updating the metric files and their provenance in run_info.json.

Field path	Meaning
model.id	Model ID used for result paths, display, and recovery.
model.params	Persisted copy of model request parameters.
model.base_url	Model API base address, saved when non-empty.
model.api_key	Redacted credential placeholder, saved when non-empty; it cannot recover the original secret.
model.api_protocol	Model API protocol name or list, saved when non-empty.
benchmark.id	Benchmark ID used to select the aggregation behavior.
benchmark.params	Effective Benchmark parameters needed to save task details and regenerate summaries.
execution	Persisted execution controls, including the exact attempts plan required for strict summary regeneration.
output.run_name	Result namespace, saved when non-empty.
output.run_id	Directory ID ultimately used for this run.

Unset fields directly under model, benchmark, execution, and output are omitted; values such as empty strings can still remain inside nested params objects. params.json does not contain the Harness, Environment, reuse settings, metadata, or complete Recipe-resolved configuration. You therefore cannot use it to reconstruct the complete evaluation configuration.

When regenerating a summary, AgentCompass reads run_info.json.request first and uses params.json to fill in missing values. The two files serve these purposes:

File	Scope	Primary purpose
run_info.json	Broader merged request, reuse source, limited execution-plan summaries, and final request state	Verify how a run was started and how it ended
params.json	Compact Model, Benchmark, execution, and output subset	Support result writes and preserve the exact attempt plan for summary regeneration

progress.json

progress.json stores the latest run state and task counts. AgentCompass replaces the snapshot with the latest state whenever a progress event occurs, so a status page or script can poll it.

Field	Meaning
run_id	This request’s run ID.
model, benchmark, harness, environment	IDs of the selected components.
status	Current run state. The file is created after the first event, so it normally starts at running and may later change to summarizing and a final request state. The internal initial value created is normally not written.
total_tasks	Number of tasks selected by the benchmark.
reused_tasks	Number of tasks reused from the source run.
pending_tasks	Tasks not yet started during execution; the count decreases at each task_started event. When the request ends, it is recalculated as total_tasks - finished_tasks, so it then also includes tasks that started but did not finish.
running_tasks	Tasks that have started but have not emitted task_finished.
finished_tasks	Reused tasks plus tasks that emitted task_finished.
completed_tasks	Tasks whose task_finished event records completed, plus reused tasks.
failed_tasks	Tasks whose task_finished event records failed. This is an execution-progress state, not the number of unsuccessful Benchmark observations.
error_tasks	Finished tasks whose highest final issue severity is error. They still count as completed_tasks; this field only reports how many of them carry errors.
warning_tasks	Finished tasks whose highest final issue severity is warning. They also count as completed_tasks.
skipped_tasks	Tasks that emitted task_finished with an explicit skipped status. Reused tasks are not rerun, but they count as completed rather than skipped.
attempts_started, attempts_finished	Evaluation attempts started and finished. Runtime retries within an attempt do not increase these counters.
partials_saved	Task-level partial results successfully persisted.
current_phase_counts	Object counting active tasks by their current phase; cleared when the request ends.
active_tasks	Object keyed by task ID that records the current state of each active task; cleared when the request ends.
elapsed_seconds	Seconds from progress-tracker creation to the latest event, rounded to three decimal places.
updated_at	Unix timestamp of the latest event, in seconds.

Each active_tasks.<task-id> object contains category, phase, attempt, and updated_at. After a task starts but before it enters a specific phase, phase is running. A missing category or attempt number is stored as null.

A task’s highest issue severity is resolved as fatal > error > warning, and every finished task counts toward exactly one severity. fatal maps to failed_tasks; error and warning map to error_tasks and warning_tasks while also being included in completed_tasks.

completed_tasks means that execution ended normally; it does not mean that the Benchmark marked the answer correct. Use task details and canonical metrics.json for Benchmark observations and aggregate values.

progress.jsonl

progress.jsonl stores the complete progress event stream. Each line is one JSON object, appended in emission order. To reconstruct a task’s phases, attempts, and retries, read this file instead of relying only on the latest snapshot.

The CLI’s --progress auto|plain|none and the SDK’s progress="auto"|"plain"|"none" control only the live terminal display. They do not disable progress.json or progress.jsonl. When an SDK caller provides a custom progress reporter, file creation depends on that reporter’s output configuration.

In the fields below, an orchestration means one launch invocation that schedules multiple evaluation requests. For a standalone request, the orchestration-related fields are null.

Fields on Every Event

Field	Meaning
run_id	Run ID.
event	Event name.
timestamp	Unix timestamp when the event was emitted, in seconds.
task_id, category	Task and category associated with the event; null for run-level events.
attempt	Evaluation attempt associated with the event, starting at 1; null when the event is not tied to an attempt.
phase	Current phase recorded by the event; null when not applicable.
status	Status recorded by the event; null when not applicable.
payload	Additional data specific to the event; an empty object when there is no additional data.
orchestration_id	Parent orchestration ID; null without orchestration context.
request_key	Unique scheduling key for the request within the orchestration; null without orchestration context.
request_name	Request name declared in the orchestration configuration; null without orchestration context.
request_index	Position of the request in the orchestration configuration, starting at 0; null without orchestration context.

All fields above are always serialized. Missing values are written as null, and payload is always an object.

Events and Event-Specific Fields

event	Fields and payload	Meaning
run_started	payload: model, benchmark, harness, environment	The request begins loading tasks.
tasks_loaded	payload.total_tasks	Task loading and selection completed.
reuse_loaded	payload.reused_tasks, payload.tasks_to_run, payload.reused_severity_counts	Reusable results were loaded and the remaining task count was determined. reused_severity_counts groups the reused tasks by highest issue severity.
task_started	task_id, category; payload.index, payload.total	The task entered scheduled execution.
phase_changed	task_id, category, optional attempt; phase	A task entered a new phase.
attempt_started	task_id, category, attempt	One evaluation attempt started.
execution_plan_resolved	task_id, category, attempt, phase: "plan"; payload is the resolved-plan summary	Plan resolution completed for the attempt; the payload matches the summary written to run_info.json.
attempt_retry	task_id, category, attempt; payload.retry, max_retries, stage, scope, matched_pattern, retry_detail	The current result was saved as retry diagnostics and will be executed again according to the matched rule.
attempt_finished	task_id, category, attempt; status is completed or failed	Attempt processing ended. Here, completed means only that the processing path returned; it does not mean the answer was correct.
partial_saved	task_id, category	A task-level result was persisted.
task_finished	task_id, category; status is completed, failed, or skipped; payload.index, payload.total, payload.severity	The task left scheduled execution. payload.severity is the highest final issue severity of that task (fatal, error, or warning), or an empty string when it has no issue.
summary_started	No additional fields	Final aggregation started.
run_finished	status is completed, failed, cancelled, or timed_out; payload.error may be present when error text is available	The request reached a terminal state.

Matching task_started and task_finished events use the same payload.index and payload.total. These values are the sequence number and task count used during scheduling; they are not task identifiers. Always use task_id to identify a task. A multi-evaluation orchestration normally preserves positions in the original selected list, so reuse can leave gaps. A single evaluation request may instead renumber the remaining tasks.

The fields in attempt_retry.payload mean:

Field	Meaning
retry	Number of runtime retries already used within the current evaluation attempt, starting at 1.
max_retries	Maximum runtime retries allowed within the current evaluation attempt.
stage	Execution stage in which the error was detected.
scope	Retry scope. attempt reruns the entire evaluation attempt; evaluate reruns only scoring or verification.
matched_pattern	Matched ERROR regex, or fatal for an unconditional FATAL retry.
retry_detail	Path to the diagnostic file containing the discarded result and error.

Current phase_changed.phase values are:

Phase	Meaning
plan	Resolve the task-level execution plan and recipes.
open_environment	Create the run environment.
prepare_task	Prepare task material in the environment.
start_harness	Start the harness session.
run_harness	Run the agent through a harness.
run_task	Let a harness-free benchmark run inference directly.
collect_artifacts	Run declared submission commands under the collection deadline. Skipped when no commands are declared; payload includes command count and source: declared_commands.
download_artifacts	Download artifacts selected by explicit paths or the default directory; progress payload includes operation, completed bytes, and remaining budget.
recover_evaluation	Validate a checkpoint and its local artifacts before opening a fresh verifier environment.
save_evaluation_checkpoint	Atomically save the clean agent context referencing the attempt artifacts.
evaluate_environment	Create a separate verification environment when the benchmark requires one.
evaluate	Score or verify the result.
save_partial	Persist the task-level result.
analyze	Update saved analysis output while re-analyzing existing results. It appears only in that workflow.

Events from concurrent tasks can interleave. Filter by task_id and attempt to follow one task. Do not assume that every task enters the same phases, and do not infer dependencies from adjacent events belonging to different tasks.

When agentcompass analysis runs, AgentCompass removes both existing progress files from the target result directory before recording the new analysis events. Without --override, the target is a newly created result copy, so the source directory is not modified.

Re-analysis keeps the original request’s run_id, but it does not rebuild run_info.json, params.json, or the run-directory log. Those files continue to describe the original evaluation request.

run.log

Each run or launch request writes framework logs to run.log. Reopening the same run directory appends to that file without overwriting earlier messages.

Logging begins after the run directory is created, before run_info.json and later runtime checks. Earlier output from the CLI or SDK is not copied into this file.

Each line uses this structure:

HH:MM:SS LEVEL    logger-name                          message

• --file-log-level controls the minimum level in the run-directory log and defaults to DEBUG; --log-level controls console output independently.
• Third-party loggers are filtered to WARNING and above by default, even when the file level is DEBUG.
• The file contains messages recorded by AgentCompass and integrated components. It does not guarantee every shell command, provider response, or internal third-party event.
• Logs are not structured results and are not inputs to summary generation, reuse, or analysis regeneration.

run_info.json and params.json redact recognized credential fields and remove underscore-prefixed runtime fields from parameter objects. This is not a general sensitive-data scanner, and it does not apply to logs.

Custom fields, free-form text, progress events, and logs may still contain paths, URLs, task data, provider information, or tracebacks. Review and remove sensitive content before sharing a run directory.

Diagnose a Failed Run

Check the files in this order to narrow down a failure:

1. Inspect progress.json for the request state and task counts. While the request is active, it also shows current phases.
2. Filter progress.jsonl by task_id to reconstruct the failing task’s last phase, attempts, and retry path. A terminal snapshot clears active tasks, so use the event stream for the last phase after the request ends.
3. Inspect run_info.json for the merged request, reuse source, and that attempt’s recipe and network-policy summary.
4. If the failure involves result persistence or summary regeneration, inspect params.json.
5. Search run.log by task ID, phase, or exception type for detailed messages and tracebacks.

Progress files describe the running process. A write failure produces a warning but does not stop the evaluation, so these files can be stale or incomplete. A forced exit can also leave run_info.json and the progress files in different states. Use persisted task details and summaries to determine the final evaluation results.

See Task Results for task-level fields, Summary and Analysis Results for aggregate metrics, and Run Controls for log-level and progress-display options.

Current Format Versions

Each record type has one supported version. These identify different structures, not alternative versions accepted by one reader.

Record	Supported schema
Run information	agentcompass.run_info.v3
Task index and shared fields	agentcompass.task.v3
Attempt checkpoint container	agentcompass.attempt_checkpoint.v1
Evaluation completion record	agentcompass.evaluation_checkpoint.v4
Artifact manifest	1