Agent Lightning Gains 14.6 SWE-bench Points Inside a Real Harness

Agent Lightning v1.0 was presented in an August 18, 2026 paper as a redesign for training agents without moving their environment loop into the reinforcement-learning engine. The Agent Lightning v1.0 paper reports that reinforcement learning on about 6,000 examples raised a Qwen3.5-9B coding-agent configuration from 41.8% to 56.4% on SWE-bench Verified—an absolute gain of 14.6 percentage points—and describes the framework as approximately 3,500 lines of core code.

The result belongs to the complete model, harness, training data and evaluation configuration, not to the lightweight library alone. An August 20 release analysis corroborates the benchmark figures and open workflow but says the trained checkpoint was not published, leaving the final model artifact unavailable for direct inspection.

What the 14.6-point result measures

The headline calculation is 56.4 minus 41.8, which equals 14.6 percentage points. Relative to the 41.8 baseline, that difference is about 34.9%, but the percentage-point formulation is the more precise comparison between two scores on the same scale.

The tested subject was a Qwen3.5-9B coding-agent configuration trained on filtered SWE-smith tasks and evaluated on SWE-bench Verified. The 56.4% score should therefore not be treated as a new universal rating for the base model, for every coding harness or for Agent Lightning deployments outside this setup.

The approximately 6,000 examples are also more demanding than a collection of static prompt-response pairs. A coding rollout can make several model calls while editing a repository, invoking tools and running tests; one rollout may consequently produce multiple training samples while retaining a single episode-level outcome.

The deployed harness stays inside the learning loop

In conventional agentic reinforcement learning, the training engine typically owns the cycle in which a model acts and an environment responds. Agent Lightning instead leaves that cycle with the existing agent harness, which continues to manage tools, context, intermediate steps and stopping decisions. Model requests and responses pass through a proxy so the trainer can observe them without taking ownership of the agent’s control flow.

This separation creates a training problem that does not appear when the engine holds one continuous token history. Reconstructed text from consecutive calls may tokenize differently because of chat templates, structured tool output or context rewriting. Calls can only be joined safely when their token histories remain continuous; otherwise, one rollout must be represented by several samples.

Advantage calculation and loss normalization then have to operate at rollout level. Without that treatment, an episode fragmented into more samples could receive greater optimization weight merely because of tokenization or harness behavior, rather than because it produced a better result.

Gateway, controller, harness and trainer form one system

The architecture can be followed as a four-part operational path. The agent harness accepts a software task and sends model calls through an OpenAI-compatible proxy. The API Gateway associates calls, events and rewards with a rollout. The Rollout Controller starts and reconciles agent executions locally or as Kubernetes Jobs. The customized trainer, built over verl, turns the recorded exchanges into samples, computes learning signals and updates the policy.

The harness is not absorbed into the framework: it remains responsible for repository interaction and tool use. The gateway connects those independently running episodes to inference and training, while the controller manages their lifecycle. This disaggregation is what permits the deployment-style harness to participate directly in post-training.

It also makes orchestration part of experimental correctness. Rollout identifiers must remain attached to the right calls and rewards, agent jobs must be reconciled after failures or delays, and a rollout’s samples must not be mixed across incompatible policy states.

The small core coordinates a much larger stack

The coding example exposes the gap between framework size and total training requirements. Microsoft’s coding-agent deployment documentation specifies Qwen3.5-9B on four B200 GPUs, Kubernetes controller mode and two machines: one prepares repository images and launches rollout Jobs, while the other runs the gateway, verl trainer and model backend.

The controller side also requires Docker, kubectl and cluster access. Each rollout runs in a repository-specific image, edits an isolated checkout, executes task tests and returns its reward to a gateway that must be reachable from both the controller and the agent pods. The GPU side still depends on the verl and vLLM training stack; the compact Agent Lightning core coordinates these components rather than replacing them.

Reward integrity adds another infrastructure requirement. The coding setup hides Git metadata and blocks routes that could reveal reference fixes, while outbound network restrictions are recommended to prevent agents from retrieving upstream answers. Rewards come from task-specific failing and passing tests inside the isolated repository environment, so weaker isolation can change the meaning of the learning signal.

Reproduction requires more than installing the package

A meaningful reproduction must preserve both the learning configuration and the execution environment. The resulting implementation checklist is broader than the framework’s core code:

1. Provision a compatible verl and vLLM backend, including the four-B200 configuration documented for the published coding workflow.
2. Prepare Kubernetes, repository-specific images, controller credentials and network routes connecting the controller and rollout pods to the gateway.
3. Route the existing harness through the rollout-specific proxy so calls, events and rewards remain associated with the correct episode.
4. Calculate rewards from isolated repository tests, conceal reference fixes and restrict outbound access before using pass rates as training signals.
5. Match the dataset filtering, rollout grouping, advantage calculation, loss normalization, training schedule and SWE-bench Verified evaluation before comparing a new result with 56.4%.

The public materials establish the architecture, scripts and reported benchmark, but they do not provide the trained checkpoint or a full accounting of elapsed training time, energy use and run-to-run variation. Independent reruns that retain the same harness isolation, reward controls and evaluation protocol are still needed to establish how reliably the 14.6-point gain reproduces beyond the reported experiment.