Task Results

Each task owns details/<state>/<readable-task-id>--<sha256>/, where <state> is normal, error or fatal according to the most severe final issue of any attempt; warning-only results are normal, and unfinished tasks stay in running. The hash is computed from the exact task ID; sanitization and truncation affect only its readable prefix.

Files

Path relative to the task directory	Purpose
task.json	Shared task identity, category, ground truth, attempt plan, and logical attempt numbers mapped to directory names such as "1": "attempt-1".
attempt-<n>/result.json	Complete result of one attempt, including its trajectory, artifact references, and retry_count.
attempt-<n>/checkpoint.json	Evaluation context and transient scheduler/download state.
attempt-<n>/artifacts/	The attempt’s actual collected files. destination is relative to this directory.
attempt-<n>/retries/	Discarded retry diagnostics and archived outputs; these do not count as independent observations.

Issues are recorded in the result, not in an _error_ filename prefix; the state directory only groups tasks. Reports, reuse, the result browser and offline analysis reconstruct the logical task record by following the index. The example below shows that reconstructed record; on disk the complete attempts payloads live in separate files.

Persisted Task Metadata

The following example shows the task-level task.json after three attempts have been allocated. New directories use attempt-<n>, where n is the logical attempt number starting at 1. During execution, the identity mapping is written first; shared result metadata is published after the attempt payloads are durable.

{
  "schema_version": "agentcompass.task.v3",
  "task_id": "example-task",
  "category": "coding",
  "ground_truth": null,
  "attempt_plan": {
    "k": 3,
    "strategy": "avg"
  },
  "attempts": {
    "1": "attempt-1",
    "2": "attempt-2",
    "3": "attempt-3"
  }
}

There is no task-level result.json. The task retry total and sparse retry_counts map are derived from each attempt’s stored retry_count. An allocated attempt without a result is not a completed observation; a terminal failure without a result payload can retain a retry-only record.

Attempt directory names must match their logical numbers: "1": "attempt-1". Cross-run reuse writes attempt-<n> directories in the new run and updates artifact references without changing the source. Retrying an attempt keeps its logical number and archives previous outputs under its own retries/ directory. See Legacy for the previous layout and reuse support.

Task Detail Example

{
  "task_id": "<task-id>",
  "category": "<category>",
  "ground_truth": "<reference-answer>",
  "attempt_plan": {
    "k": 3,
    "strategy": "avg"
  },
  "retry_count": 2,
  "retry_counts": {
    "1": 2
  },
  "attempts": {
    "1": {
      "status": "completed",
      "metrics": {
        "correct": true,
        "reward": 0.82
      },
      "final_answer": "<answer>",
      "trajectory": {},
      "issues": [],
      "artifacts": {},
      "analysis_result": {},
      "meta": {
        "benchmark": {
          "...": "..."
        },
        "harness": {
          "...": "..."
        }
      }
    }
  }
}

Every task detail uses this fixed standard shell. Empty values remain explicit as null, "", or {}. The ... entries only indicate component-specific content; a namespace with no content is written as {}.

Task-Level Fields

Field	Required?	Meaning
task_id	Yes	Stable, non-empty task ID supplied by the Benchmark. Leading or trailing whitespace is invalid.
category	Yes	Benchmark category used for grouped aggregation, or null when none is assigned.
ground_truth	Yes	Task-level reference data; it may be null for a hidden verifier. It is not repeated inside attempts.
attempt_plan	Yes	The k and strategy used to produce this record.
retry_count	Yes	Sum of retries consumed across all logical attempts.
retry_counts	Yes	Sparse map from string attempt number to retries consumed by that attempt. Its values sum to retry_count.
attempts	Yes	Non-empty map keyed by canonical positive integer strings such as "1".

Attempt-Level Fields

Every attempt writes all standard fields. Empty values remain present so task-detail consumers see the same field set for every attempt.

Field	Required?	Meaning
status	Yes	completed, skipped, run_error, eval_error, run_error_or_eval_error, cancelled, or interrupted. A completed attempt is not necessarily successful.
metrics	Yes	Benchmark observations keyed exactly as declared by its Metric Contract. Values are JSON booleans or finite numbers.
final_answer	Yes	Text, patch, or structured answer produced by the Model or agent; null when unavailable.
trajectory	Yes	Harness-normalized interaction trace, or {} when unavailable.
issues	Yes	Structured unresolved issues; [] when empty.
artifacts	Yes	Integration-specific artifacts or indexes, or {} when empty.
analysis_result	Yes	Analyzer output keyed by analyzer family, or {} when empty.
meta	Yes	Namespaced extension data that is useful but is not a generic Benchmark metric.

meta Namespaces

Namespace	Owner and examples
meta.benchmark	Benchmark-specific grader diagnostics, component scores, or special fields that are not generic observations.
meta.harness	Harness diagnostics and telemetry, such as token or latency counters.

Both namespaces are always present and use {} when empty. Their internal component-specific keys are not part of the common task-detail schema.

Resolved Environment, Recipe, and network plans are stored in run_info.json.resolved_execution_plans; see Run Records and Diagnostics.

Trajectory Shape

When present, trajectory uses the AgentCompass ACTF_v1.0 shape:

Field	Meaning
schema_version	Trajectory schema version.
steps	Ordered model, tool, Environment observation, and timing steps.
started_at, finished_at	Complete trajectory timestamps.

Common step fields include step_id, prompts, assistant content, tool calls, observations, timestamps, and metric token/timing values. Harnesses can omit data they do not produce.

Analysis Results

analysis_result.<analyzer-family> can contain is_badcase, score, details, error, and extra. These are analyzer diagnostics and do not alter attempt.metrics or its status. Run-level analysis files combine analyzer output across attempts separately from Benchmark metric aggregation; see Summary and Analysis Results.

Retry Details

When a failure matches the retry policy and budget remains, AgentCompass writes agentcompass.retry.v1 diagnostics before rerunning the current logical attempt:

{
  "schema_version": "agentcompass.retry.v1",
  "task_id": "<task-id>",
  "attempt": 3,
  "retry": 1,
  "max_retries": 2,
  "stage": "evaluate",
  "scope": "evaluate",
  "matched_pattern": "fatal",
  "issues": [{"severity": "fatal", "phase": "evaluate", "code": "judge_failed", "message": "HTTP 503"}],
  "discarded_result": {}
}

attempt identifies the unchanged logical attempt; retry is its one-based retry number. stage locates the failure, while scope says whether the runtime repeats the complete attempt or only evaluation. discarded_result is diagnostic and is not required to match the strict task-detail shape.

Completed sibling attempts remain checkpointed. For example, a retry in attempt 3 does not rerun attempts 1 and 2. The final task detail records the consumed retry counts, while only the terminal attempt payload contributes observations.

Sensitive Content

AgentCompass redacts recognized credential fields before persistence, but answers, trajectories, errors, artifacts, and integration-specific metadata can still contain sensitive task content. Review them before sharing a run directory.

Related Pages

• Metrics and Aggregation
• Summary and Analysis Results
• Run Records and Diagnostics
• Run Controls