Save and Prepare Artifacts

Save files produced inside the task Environment so you can inspect them locally or restore them for evaluation. AgentCompass normally uses the paths and preparation commands declared by the Benchmark; you only need overrides when changing what is saved or how submissions are prepared.

Answers, trajectories, metrics, and runtime logs are part of the normal evaluation results. The settings here control additional files transferred from the task Environment.

Save Files Locally

To save the Benchmark’s declared files even when evaluation reuses the execution Environment, add this to your configuration:

execution:
  save_artifacts: true

Load the file with --config, or add --execution-params '{"save_artifacts":true}' to your existing run command. The Benchmark must support artifact collection. Saved files appear under the attempt’s local artifacts/ directory; see the result directory layout.

Saving files does not retain a running sandbox. Use --keep-environment when you need to inspect the Environment itself.

Understand Saving and Restoration

The Environment’s evaluation mode determines the default behavior:

Evaluation mode	Default saving behavior	How scoring accesses the files
fresh	Save declared artifacts; saving is required.	Restore them into a new evaluation Environment.
reuse	Save only when save_artifacts=true.	Read files already in the execution Environment.
none	Save only when save_artifacts=true.	No evaluation Environment is created.

Omitting save_artifacts or setting it to null uses the mode’s default. Disabling downloads does not skip declared preparation commands or Harness result collection. See evaluation environments for mode configuration.

Customize Submission Files

Only override paths or preparation commands when you know what the Benchmark’s evaluator expects. Both lists replace the Benchmark’s declarations, so include every required original entry in your replacement.

Field	Omitted or null	Explicit list
artifacts	Inherit declared paths.	Replace extra paths; /logs/artifacts/ is always included. [] selects only that directory.
artifact_collect	Inherit preparation commands.	Replace commands in execution order. [] clears the list.

For example, if your Benchmark expects /app/submission.tar.gz and the agent writes its submission under /app/submission, prepare and save that archive:

execution:
  save_artifacts: true
  artifacts:
    - source: /app/submission.tar.gz
      destination: submission.tar.gz
  artifact_collect:
    - command: "tar -czf /app/submission.tar.gz -C /app submission"
      timeout_seconds: 120

The command runs after agent execution. It creates the archive in the task Environment, then AgentCompass saves it locally as artifacts/submission.tar.gz. Fresh evaluation restores it to its original source path.

Use absolute sandbox paths for source and relative local paths for destination. An optional exclude list filters files by pattern. Preparation commands run in the Environment’s default working directory; use absolute paths or an explicit cd for task-specific locations. Missing sources are recorded in the artifact manifest without failing collection.

Limit Preparation and Transfer

Change these limits when submission preparation or transferring large files exceeds the defaults:

execution:
  artifact_collect_timeout_seconds: 300
  artifact_limits:
    timeout_seconds: 1800
    max_mb: 32768
    max_files: 100000

This allows 300 seconds for all preparation commands together and 1,800 seconds for each batch transfer, with a 32 GiB size limit. Each command still uses its own timeout_seconds, as in the submission example above.

Field	Default	What it limits
artifact_collect_timeout_seconds	null	Total duration of all preparation commands; null sets no group deadline.
artifact_collect[].timeout_seconds	60	One preparation command.
artifact_limits.timeout_seconds	600	One batch download or restore, shared by all declared files.
artifact_limits.max_mb	16384	Total size per collection or restore, in MiB; also caps each transfer archive, including overhead.
artifact_limits.max_files	100000	File and directory entries per collection or restore.

For preparation, the first of the command or group deadlines wins. A batch download and a later restore each start a new transfer clock; at the defaults, each can use 600 seconds. Skipped transfers consume no budget. These limits do not extend the overall evaluation timeout.

Preparation or transfer failure preserves validated files and execution diagnostics, but skips grading and creation of a resumable evaluation checkpoint for that attempt. Check the task’s issues and transfer progress before increasing a limit.