Dion3 Cuts Muon Optimizer Time by Up to 6×—With Parallelism Limits

Noah Amsel and seven co-authors introduced Dion3 in an arXiv paper submitted on August 12, 2026, finding that their combined optimizer stack matched or improved the loss achieved by Muon while cutting measured optimizer-step time by up to sixfold. The timed interval excludes the forward and backward passes, so the result does not mean that an entire training run becomes six times faster.

Deployment support is narrower than the paper’s description of Dion3 as a drop-in Muon replacement. Microsoft’s public Dion implementation and compatibility table currently list Dion3 for a single device, PyTorch DDP and FSDP2, but not for FSDP2 combined with tensor parallelism; the repository requires PyTorch 2.7 or newer and DTensor-based distributed execution.

The 6× result measures optimizer overhead

The benchmark isolates GPU work performed inside the optimizer step: momentum processing, orthogonalization, parameter updates and optimizer communication. Model execution, gradient calculation and input handling sit outside that boundary. This makes the result an optimizer-latency benchmark rather than an end-to-end throughput measurement.

The practical training gain therefore depends on the share of wall-clock time that Muon consumes before Dion3 is introduced. If forward and backward computation dominate a workload, even a large reduction in optimizer latency will produce a much smaller change in total step time. Work already hidden by overlap with backward execution also offers no additional visible saving.

The distinction matters most when comparing optimizer implementations. Dion3 addresses Newton–Schulz computation and the communication needed when matrix weights are sharded, but it does not accelerate attention, feed-forward layers or gradient computation. Infrastructure teams need a profile of exposed optimizer time before treating the paper’s upper-bound result as a capacity-planning input.

The tests cover several model scales but remain author-run

The paper’s performance work includes single-accelerator tests on an NVIDIA GH200 and distributed FSDP tests across four GH200s, while its model-quality comparisons cover language models at 3B, 4B, 7B and 14B parameters trained on 10 billion ClimbMix tokens. The largest headline reduction comes from the complete Dion3 configuration rather than a single kernel or algorithmic substitution.

Results vary with matrix shape, sharding, communication behavior and the fraction of the momentum matrix selected for orthogonalization. That variation is why “up to” is essential: the sixfold figure is the best measured optimizer-step result in the published experiments, not a uniform ratio across every tested model or distributed layout.

The loss comparisons are encouraging but bounded. They show that the authors’ fractional-update configurations preserved or improved optimization quality against their tuned Muon-family baselines at the tested scales. They are not yet independent reproductions across different datasets, architectures, accelerator generations or production training stacks.

Dion3’s fastest path is a combined systems stack

Dion3 obtains its reduction from several changes that compound. Gram Newton–Schulz performs the iterative calculation on a smaller symmetric Gram matrix; custom symmetric kernels accelerate that calculation; megabatching reduces collective-communication rounds; and the update rule orthogonalizes only a selected fraction of the momentum matrix.

The fractional update is the part that changes the optimization trajectory. Error feedback carries information omitted from one update into later steps, while per-neuron normalization is inherited from NorMuon. A full-fraction configuration reduces to Muon in exact arithmetic, but the faster fractional configurations should not be described as mathematically identical to a complete Muon update.

The optional Gram Newton–Schulz kernel package specifies NVIDIA Hopper H100 or Blackwell B200/B300 hardware, PyTorch 2.7.1 or newer and CUDA 12.9 or newer. Those kernels can improve Muon’s orthogonalization independently, but installing them alone does not reproduce Dion3’s full benchmark because fractional updates and distributed megabatching are separate parts of the stack.

The compatibility matrix rules out tensor parallelism

The published implementation yields a clear deployment split:

• Single device: supported.
• PyTorch DDP: supported when the optimizer receives the DDP process group.
• PyTorch FSDP2: supported through a one-dimensional sharded device mesh.
• FSDP2 plus tensor parallelism: unsupported for Dion3; the legacy Dion optimizer is the repository’s implementation for that combination.

This is a structural limitation rather than an omitted launch example. Dion3 accepts one sharded mesh dimension, while an FSDP2-plus-TP system introduces another independently sharded dimension. A training stack that depends on tensor parallelism cannot replace Muon with the released Dion3 implementation without extending its distributed layout handling.

Integration also requires deliberate parameter grouping. Dion3’s matrix update applies to appropriate two-dimensional weights, while vectors and other scalar-style parameters need an auxiliary algorithm such as AdamW or Lion. Embeddings and language-model heads require particular care because their tensors can look like ordinary matrices even when they should not receive the same optimizer treatment.

Dion3 is a conditional Muon replacement

Dion3 is currently a plausible replacement for Muon when the workload runs on one device, DDP or one-dimensional FSDP2 and profiling identifies exposed optimizer work as a material bottleneck. Adoption also assumes compatible PyTorch distributed primitives, correct parameter groups and access to the optional kernel dependencies when the optimized Gram Newton–Schulz path is required.

The case is weaker for systems whose training time is dominated by forward and backward computation, and it stops at the current tensor-parallel boundary. As of the implementation reviewed for this release, broader end-to-end measurements, independent reproductions and Dion3 support for FSDP2-plus-TP remain the missing evidence needed to generalize the headline benchmark to large hybrid-parallel training clusters.

Also read:

• Oligo Raises $60M as AI Shrinks the Time Between Bug and Exploit
• YouTube’s Total Dominance: For the First Time Ever, It Has Overtaken Netflix in Daily Viewing Time
• Ai2’s SERA Is Still Open—but Its $400 Cost Claim Has Limits