Network Policy

Choose, configure, verify, and troubleshoot baseline, run, and evaluation network policies.

AgentCompass can control outbound network access separately for trusted environment setup, the complete untrusted run boundary, and formal verification. Use these controls to reproduce an official benchmark policy, prevent an agent from retrieving external solutions, or allow only the endpoints required by a controlled evaluation.

Start with the policy documented by the selected benchmark. Changing network access can change both task difficulty and result comparability, so an alignment run should not silently broaden or narrow the official setting.

Choose a Policy for Each Phase

Network policy is resolved independently for every task. A Benchmark loader can declare the policy fields on each TaskSpec, so two samples in the same run can use different policies. Values passed through --env-params have run-wide scope and explicitly override the corresponding phase for every selected sample:

Field	Protected phase	Common choice
baseline_network_policy	Agent Environment startup, benchmark preparation, and trusted harness setup; common request override for fresh startup	public by default; keep it public when packages or a harness executable must be installed
evaluation_baseline_network_policy	Independent fresh-verifier startup and reset	Inherits the agent baseline only when no independent verifier baseline is declared
run_network_policy	Agent or harness rollout, Harness close, artifact collection, and the remaining lifetime of the agent Environment	public by default; often no-network for isolated coding tasks
evaluation_network_policy	Formal evaluation in the reused task Environment or a fresh evaluation Environment	public by default; use no-network for local tests, or allowlist/public when grading requires an external service

These are not four successive phases. Shared evaluation has one environment baseline plus run/evaluation overrides. Fresh evaluation has two environments, each with its own baseline, and still only two execution-phase overrides. Harbor maps [environment].network_mode to the agent baseline and [verifier.environment].network_mode to the separate verifier baseline; [agent].network_mode and [verifier].network_mode remain the phase overrides. Request-level evaluation_baseline_network_policy requires fresh mode, and overrides a common request baseline before falling back to the task’s verifier baseline or the inherited agent baseline. An explicit empty Harbor verifier environment uses its own default public baseline.

The existing phase fields are resolved with this precedence:

Priority	Source	Scope
1	Explicit Environment override, including --env-params	Every selected task in the run
2	TaskSpec sample policy populated by the Benchmark loader	One task
3	Compatible Recipe policy	Each matching task
4	Runtime default	Each omitted phase independently becomes public

Omitting a field means “continue to the next source”; if no source supplies that phase, it becomes public. Explicitly passing public and leaving a phase unset therefore produce the same effective policy when no other source supplies one. Use --env-params when intentionally forcing one policy across a run. To preserve official per-sample behavior, omit those overrides and let the Benchmark load its task policy. A compatible Recipe may validate execution-required endpoints, but it does not change the resolved policy; the effective policy and applied_recipes record the resulting decision.

Harness setup happens before run_network_policy is applied. This lets a trusted harness install its runtime under the baseline policy and then execute the untrusted agent under a stricter policy. Network phases are function-level trust boundaries, not a separate policy for every progress stage.

Understand Function-Level Boundaries

AgentCompass applies the same evaluation rule to both evaluation Environment modes: only the complete benchmark.evaluate() call uses evaluation_network_policy. The modes differ because fresh creates another Environment, while reuse evaluates in the agent Environment. The evaluate_environment progress phase in a fresh run is not a separate Benchmark hook; its setup boundary is the evaluation provider’s open() call.

The tables below show the active policy for Environment operations. Provider control-plane requests made by the AgentCompass host remain outside sandbox network enforcement. A function can contain several internal operations; the runtime does not split those operations into finer policy scopes.

Shared Environment

With reuse, the same Environment remains active while its lifecycle functions move through the baseline, run, and evaluation policies:

Order	Runtime function boundary	Environment	Active policy	Condition
1	environment_provider.open()	Shared	baseline_network_policy	Always
2	benchmark.prepare_task()	Shared	baseline_network_policy	Always
3	harness.start_session()	Shared	baseline_network_policy	Harness-based Benchmark only
4	harness.run_task() or benchmark.run_task()	Shared	run_network_policy	One inference path is selected
5	harness.close_session()	Shared	run_network_policy	When a Harness session was started
6	Declared collection commands, then download_artifacts()	Shared	run_network_policy	Preparation requires declared commands; download requires resolved paths and effective save_artifacts=true
7	benchmark.evaluate()	Shared	evaluation_network_policy	Always; receives the same Environment
8	environment_provider.close()	Shared	baseline_network_policy	Unless the Environment is retained

Immediately before step 7, the runtime switches directly from run_network_policy to evaluation_network_policy, calls the entire benchmark.evaluate() function, and restores baseline_network_policy in finally. This keeps the evaluation policy scoped to the evaluation function without an intermediate baseline transition.

Separate Environment

fresh completes the agent Environment lifecycle first, then creates a separate evaluation Environment:

Order	Runtime function boundary	Environment	Active policy	Condition
1	environment_provider.open()	Agent	baseline_network_policy	Always
2	benchmark.prepare_task()	Agent	baseline_network_policy	Always
3	harness.start_session()	Agent	baseline_network_policy	Harness-based Benchmark only
4	harness.run_task() or benchmark.run_task()	Agent	run_network_policy	One inference path is selected
5	harness.close_session()	Agent	run_network_policy	When a Harness session was started
6	Declared collection commands, then download_artifacts()	Agent	run_network_policy	Preparation requires declared commands; download requires resolved paths and effective save_artifacts=true
7	environment_provider.close()	Agent	run_network_policy	Unless the agent Environment is retained
8	evaluation_provider.open()	Evaluation	evaluation_baseline_network_policy	Always in fresh mode
9	benchmark.evaluate()	Evaluation	evaluation_network_policy	Always; receives the fresh Environment
10	evaluation_provider.close()	Evaluation	evaluation_baseline_network_policy	Unless the evaluation Environment is retained

Step 8 is the conditional evaluation Environment setup introduced by selecting fresh; it is not an optional Benchmark hook. The runtime switches to the evaluation policy only after evaluation_provider.open() completes, restores the evaluation Environment baseline immediately after benchmark.evaluate(), and then releases or retains that Environment.

Runtime executes declared collection commands directly, then collects existing files when storage is enabled. Without commands, it skips preparation. Commands and downloads remain inside the run trust boundary; they do not create another network phase.

Network Modes

Each phase accepts one of four modes:

Mode	Behavior	Use it when
public	Allows normal outbound access.	Setup needs package registries, source downloads, or unrestricted external services.
no-network	Blocks outbound network access.	The task must be solved only from the provided workspace and local tools.
allowlist	Allows only explicitly listed hosts, addresses, or networks.	The agent needs a model endpoint or controlled service without general internet access.
denylist	Allows normal outbound access except for explicitly denied targets.	A compatible provider needs broad access with a small set of blocked endpoints.

Use a string for public or no-network:

export MODEL_NAME=""

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --env docker \
  --env-params '{
    "baseline_network_policy":"public",
    "run_network_policy":"no-network",
    "evaluation_network_policy":"no-network"
  }'

Use an object for an allowlist or denylist:

{
  "baseline_network_policy": {
    "network_mode": "allowlist",
    "allowed_hosts": [
      "pypi.org",
      "registry.npmjs.org:443",
      "files.pythonhosted.org",
      "*.example.com",
      "203.0.113.10",
      "203.0.113.0/24"
    ]
  },
  "run_network_policy": "no-network",
  "evaluation_network_policy": "no-network"
}

{
  "run_network_policy": {
    "network_mode": "denylist",
    "denied_hosts": [
      {"host": "registry.npmjs.org", "port": 443}
    ]
  }
}

Host values must be hostnames, leading-wildcard hostnames, IP addresses, or canonical CIDR ranges. Do not include a URL scheme, path, whitespace, or an embedded wildcard such as api.*.example.com. Both allowlists and denylists accept a bare host such as www.example.com, a host:port shorthand such as www.example.com:443, or a target object such as {"host": "www.example.com", "port": 443}. Bare hosts retain all-port semantics; port values must be integers from 1 through 65535. Use [2001:db8::1]:443 when adding a port to an IPv6 address. Providers that cannot enforce a port-specific target reject it.

Tell the Agent About Rollout Restrictions

By default, each Harness appends an English network restriction statement to its final user instruction when the effective run_network_policy is no-network, allowlist, or denylist. This prevents an agent from treating an intentional restriction as a transient network failure and repeatedly retrying blocked operations. A public policy does not add a statement.

The statement is based on the provider-resolved policy, not only the requested value. Recipe-added targets therefore appear in an allowlist statement, and allowlist or denylist targets are expanded from the effective policy. A provider that cannot enforce the requested policy rejects the run before rollout instead of injecting a misleading statement.

<IMPORTANT>
  Network Restriction Statement:
  Ignore any duplicate or conflicting declarations about network access elsewhere in the instructions; follow this statement instead.
  This statement has the highest priority for network access because it explicitly reflects the effective network policy applied to the task sandbox.
  Outbound network access from the task environment is restricted to the following targets:
  - pypi.org
  - registry.npmjs.org:443
  All other outbound targets are blocked. Do not repeatedly retry access to blocked targets.
</IMPORTANT>

no-network uses the same wrapper without a target list:

<IMPORTANT>
  Network Restriction Statement:
  Ignore any duplicate or conflicting declarations about network access elsewhere in the instructions; follow this statement instead.
  This statement has the highest priority for network access because it explicitly reflects the effective network policy applied to the task sandbox.
  Outbound network access from the task environment is disabled. Use only resources already available in the environment. Do not repeatedly retry operations that require external network access.
</IMPORTANT>

For denylist, the effective denied targets replace the list:

<IMPORTANT>
  Network Restriction Statement:
  Ignore any duplicate or conflicting declarations about network access elsewhere in the instructions; follow this statement instead.
  This statement has the highest priority for network access because it explicitly reflects the effective network policy applied to the task sandbox.
  Outbound network access from the task environment is available except for the following blocked targets:
  - example.com:443
  Do not repeatedly retry access to these blocked targets.
</IMPORTANT>

The Harness injects this block only into the rollout copy of the prepared input. Artifact collection and benchmark.evaluate() continue to receive the original input. Standard Harness paths support plain prompts, structured user messages, multimodal user messages, and the JSON-encoded messages accepted by openai_chat. The harness-free TauBench inference path does not currently consume this notice.

Disable injection through the selected Harness configuration:

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --harness-params '{"inject_network_restriction_notice":false}'

For a persistent default, set inject_network_restriction_notice: false under the selected harnesses.<id> entry in a YAML or JSON configuration file. Python callers pass the same field in harness_params. This setting changes only the Harness input notice; it does not change or disable network-policy enforcement.

Pass Phase Policies from the CLI

Pass the three runtime policies as JSON fields in --env-params:

export MODEL_NAME=""

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --env docker \
  --env-params '{
    "baseline_network_policy":"public",
    "run_network_policy":"no-network",
    "evaluation_network_policy":{
      "network_mode":"allowlist",
      "allowed_hosts":["judge.example.com:443"]
    }
  }'

The runtime applies those CLI values as follows:

CLI field	Runtime behavior
baseline_network_policy	Creates the agent Environment under this policy (and the fresh verifier unless specifically overridden), keeping it active through benchmark.prepare_task() and harness.start_session().
run_network_policy	Switches to this policy before inference and keeps it active through Harness close, collection commands, and artifact download.
evaluation_network_policy	Switches to this policy for the complete benchmark.evaluate() call, then restores the baseline.

The same values can be supplied as environment_params through the Python API, or by a Benchmark loader on TaskSpec for sample-level policy.

Select the Narrowest Practical Policy

Use this decision sequence:

1. Check the benchmark page for an official or recommended policy.
2. Identify where the harness is installed and where it calls the model API.
3. Keep the baseline public if the sandbox must install a package or executable; otherwise prefer an allowlist or a prebuilt image.
4. Set the run phase to no-network when the task should use only local evidence.
5. Include any endpoints needed by Harness close or artifact collection in the run policy; do not broaden it between rollout and verification.
6. Add only the exact model, search, judge, package, or artifact hosts required by a network-dependent phase.
7. Run one task and inspect the resolved execution plan before scaling.

The Python packages used by the AgentCompass driver are installed outside the task sandbox and are not controlled by these policies. Packages or CLI tools installed by harness.start_session run inside the environment and therefore use the baseline policy. If the baseline must also be no-network, put those dependencies in the task image or snapshot first.

Whether a model endpoint needs to be allowlisted depends on where the harness makes its request:

• A local harness process calls the model from the AgentCompass host, outside the task environment policy.
• A harness running inside the sandbox needs the model endpoint in the run-phase allowlist.
• Some benchmark recipes, including DeepSWE recipes, infer the resolved model endpoint. Do not assume every custom benchmark or external recipe does so; inspect the resolved plan.
• The meaning of run_network_policy is independent of where it came from. DeepSWE validates that its required model endpoints are permitted and fails during planning if the effective task or CLI policy blocks them; it never broadens that policy. A remote Harness therefore needs an explicit CLI allowlist containing the model host; a local Harness can keep no-network.

The same distinction applies to judge and search services. A request made by the AgentCompass driver is outside the sandbox policy; a request made by a process inside the task or verifier environment must be allowed in that phase.

Provider Support

Provider	Modes	Dynamic phase changes	Important limits
host_process	public only	No	It cannot provide sandbox network isolation.
docker	public, no-network, allowlist, denylist	Yes	Phase switching requires a bridge-style network. Allowlist and denylist control use an egress proxy sidecar.
daytona	public, no-network, allowlist	Yes	Supports domains, wildcard domains, IPv4 addresses, and IPv4 CIDRs; domain and network entries cannot be mixed.
modal	public, no-network, allowlist	Yes	Supports domain, IPv4, and IPv6 entries; dynamic switching requires a compatible Modal SDK.

Daytona accepts at most 20 domain entries or 10 IPv4 network entries. Docker cannot use dynamic phase policies with network set to none, host, or container:<id>. Its default egress proxy image is python:3.12-alpine; make sure the Docker daemon can pull it or pre-pull it on an offline host.

Outbound policy accepts only the shared fields above. Provider-native block-all and allowlist parameters are rejected; adapters generate those API options from the resolved shared policy.

Docker is currently the only documented public provider that enforces denylist. Other public providers reject that policy instead of silently broadening it to public.

Verify the Effective Policy

Run one known task with persistent debug logs:

export MODEL_NAME=""

agentcompass run <benchmark> <harness> "$MODEL_NAME" \
  --env docker \
  --benchmark-params '{"sample_ids":["<task-id>"]}' \
  --env-params '{
    "baseline_network_policy":"public",
    "run_network_policy":"no-network",
    "evaluation_network_policy":"no-network"
  }' \
  --task-concurrency 1 \
  --max-retries 0 \
  --log-level INFO \
  --file-log-level DEBUG

The run log records baseline_network_mode, run_network_mode, and evaluation_network_mode when each task execution plan is built. Per-task details retain the resolved policies and applied_recipes. Verify those effective values rather than relying only on the original command, because a Benchmark Recipe may add an inferred endpoint or provider adaptation.

For an adversarial isolation test, ask the agent to access a known external URL and confirm both outcomes:

• the request fails during the restricted run phase; and
• the same environment can still perform the trusted setup work allowed by its baseline policy.

For supported terminal trajectories, NetworkOperationAnalyzer can summarize commands such as curl, wget, package installation, or git clone. It observes agent behavior but does not enforce the policy and cannot replace provider transition logs.

A failed application request is not sufficient evidence by itself. It may be caused by DNS, credentials, or an unavailable service. Confirm the resolved policy and provider transition logs as well.

Troubleshoot Network Failures

Symptom	Likely cause	Action
Harness installation fails during setup	The baseline is restricted or required registry hosts are absent.	Use public, extend the baseline allowlist, or preinstall the harness in the image.
Model calls fail only after harness setup	The harness calls the model inside the sandbox, but the run policy blocks the endpoint.	Add the endpoint hostname to run_network_policy.allowed_hosts.
Artifact collection fails after a successful rollout	A close or collection hook needs an endpoint blocked by the run policy.	Add only the required artifact endpoint to run_network_policy, or keep collection local.
An allowlisted URL is still blocked	A redirect, artifact CDN, authentication host, or DNS target is missing.	Inspect the request chain and add exact required hosts; avoid broad wildcards.
Daytona rejects the allowlist	Domains and IPv4 networks were mixed, an IPv6 entry was used, or the provider entry limit was exceeded.	Use one supported entry family and stay within provider limits.
Docker rejects phase switching	The selected Docker network is not bridge-style.	Remove the custom network or use a bridge network.
Docker egress proxy cannot start	The proxy image is unavailable, Docker lacks permission, or the proxy startup timeout is too short.	Pull the image while online, verify Docker access, or increase allowlist_proxy_start_timeout.
Modal or Daytona reports that dynamic switching is unsupported	The installed provider SDK lacks the required runtime API.	Upgrade the provider SDK through the AgentCompass installation and retry one task.
Verification fails after a successful rollout	The verifier needs a local dependency or external service blocked by its policy.	Prefer a prebuilt verifier; otherwise configure only the verifier phase appropriately.

Related Pages

• Choose an Environment
• Sandbox Resource Limits
• Recipes