Benchmark an AWS Agent Without Letting It Touch Production

The safest way to run aws-bench is to create a dedicated non-production management account, use it to provision disposable member accounts, and let the agent operate only inside those test accounts. Keep production profiles, access keys, workloads, and organization-management credentials out of the benchmark host and containers.

Begin with the quickstart dataset, use the same environment and dataset identifiers throughout the lifecycle, and clean up as soon as the run is finished. aws-bench deploys real AWS infrastructure, so account isolation limits the blast radius; it does not make the evaluation free or eliminate the need to verify credentials and teardown.

Establish the account boundary first

AWS describes aws-bench as a research preview for measuring agents on investigation, troubleshooting, and infrastructure-creation tasks in its July 2026 announcement. Treat that status and the use of live cloud resources as reasons to separate the benchmark from production rather than attaching it to an existing production organization.

Create an AWS account specifically for benchmark orchestration. It becomes the management account for a test organization and should contain no production workloads; the disposable member accounts beneath its test Organizational Unit hold the scenario resources.

• Operator host: runs the CLI with the dedicated management profile.
• Management account: creates and controls the test organization, Organizational Unit, accounts, and quotas.
• Scenario account: contains the real infrastructure for one scenario.
• Agent container: runs sandboxed with credentials scoped to that scenario account.

The isolation path is operator host → non-production management account → test Organizational Unit → disposable scenario account → sandboxed agent. Before initialization, run aws sts get-caller-identity and compare the returned account ID with the dedicated benchmark account; stop if it resolves to production.

Check prerequisites and credential precedence

The official Getting Started guide specifies macOS or Linux, Python 3.12 or newer, uv, Docker with Compose v2 and buildx 0.17.0 or newer, plus an AWS account allowed to create an organization and member accounts. It also states that the management account holds no test resources and that the agent operates in a test account, never in the management account.

1. Configure a dedicated AWS CLI profile, preferably through IAM Identity Center: aws configure sso --profile my-awsbench-profile.
2. Remove exported credentials that could override the profile: unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN.
3. Select the profile with export AWS_PROFILE=my-awsbench-profile, then run aws sts get-caller-identity.
4. Set both AWS_REGION=us-east-1 and AWS_DEFAULT_REGION=us-east-1.
5. Clone the repository, enter its directory, run uv sync, and confirm the CLI with uv run aws-bench --help.

The unset step matters because environment access-key variables take precedence over AWS_PROFILE. For Bedrock-backed runs, stale keys can cause env creds to mint its service-specific token against the wrong account. Never pass the management account’s general AWS credential chain into the agent container.

Provision the quickstart environment

The quickstart is a nine-task, single-scenario smoke test. Use it to confirm account creation, quota handling, deployment, agent execution, verification, and cleanup before allocating a larger environment.

1. Initialize accounts and quota requests with uv run aws-bench env init --env-name awsbench-env -d aws-bench-quickstart --wait-for-quotas.
2. If quota approval is still pending, inspect state with uv run aws-bench env show --env-name awsbench-env.
3. Deploy the scenario using uv run aws-bench env setup --env-name awsbench-env -d aws-bench-quickstart.
4. For a Bedrock-backed agent or judge, recheck the active identity and run eval $(uv run aws-bench env creds --eval).

Use the identical -d value for initialization, setup, execution, verification, and cleanup. Initialization creates the organization structure and accounts; setup builds scenario containers and deploys CDK stacks. The billable window begins when real resources are deployed, so do not leave a completed setup idle.

Run and record a reproducible evaluation

Run uv run aws-bench run --env-name awsbench-env -d aws-bench-quickstart -a <agent> -m <model-id> --yes. When the verifier or agent needs Bedrock, add --ve AWS_BEARER_TOKEN_BEDROCK=$AWS_BEARER_TOKEN_BEDROCK; the flag passes that provider credential into the verifier or agent container rather than exposing the host’s general management credentials.

Record the repository revision, environment name, dataset name and version, agent, model, region, and run arguments with the result. An unversioned dataset name selects the latest registry version, while name@version pins the task bundle for comparisons over time.

The dataset catalog lists 9 quickstart tasks, 78 basic tasks, and 47 advanced tasks. It also permits individual scenarios to run as separate datasets, which is the narrower next step when you want to constrain deployed services, quotas, runtime, and cost.

Interpret the verifier, not just the score

Each trial receives 1.0 for a pass or 0.0 for a failure. Inspect jobs/<timestamp>/agent/agent-output.txt, verifier/test-stdout.txt, verifier/reward.json, and verifier/reward-details.json; aggregated results appear in jobs/<timestamp>/result.json.

Read-only introspection tasks use an LLM judge to compare the agent’s answer with stored ground truth. Mutation tasks use a programmatic boto3 verifier against live AWS state. A pass therefore means that the configured verifier accepted that trial, not that every aspect of the agent’s reasoning or infrastructure change was correct.

Repeated LLM-judged trials can vary, so compare equivalent dataset versions, models, agents, credentials, regions, and run settings. If a result appears contaminated, run uv run aws-bench env verify --env-name awsbench-env -d aws-bench-quickstart; detected drift or new resources can produce false passes, resource collisions, or avoidable failures.

Clean up resources, then decide whether to close accounts

A completed run automatically resets scenarios toward their post-setup baseline, but it does not remove the deployed infrastructure. End the session with uv run aws-bench env cleanup --env-name awsbench-env -d aws-bench-quickstart --yes. Cleanup removes deployed resources while retaining the test accounts; it is the documented cost-control step.

1. Preserve the job directory and run configuration outside the disposable accounts.
2. If investigating suspected contamination, run env verify before cleanup and retain its output.
3. Run cleanup with the exact environment name and dataset used for setup.
4. Review the complete cleanup output and inspect any account named in a failure for residual billable resources.
5. If the environment will not be reused, run uv run aws-bench env terminate --env-name awsbench-env to close its test accounts.

Keep cleanup and termination distinct. Cleanup removes scenario infrastructure and stops the intended resource charges; termination additionally closes the disposable accounts. Do not treat a successful benchmark score as evidence of successful teardown—the cleanup output and the resulting account state are separate checks.

Also read:

• How to Appeal a Disabled Instagram Account Without Following the Wrong Recovery Path
• Figma’s Design Agent Is Open Beta—Paid Access Still Has Boundaries
• How to Use TikTok's Commercial Music Library Without Licensing the Wrong Track