Aligning Only Three Rounding Points: BF16 Large-Model Reinforcement Learning No Longer Collapses

On October 23, 2026, researchers Yijing Ke, Tianle Cai, and Ruosong Wang published a paper on arXiv pointing out that the root cause of collapse in large language model reinforcement learning training lies in an engineering detail: the inference stack and the training stack round at inconsistent times in three mathematical operations. Fixing these three points is enough to stabilize reinforcement learning training under BF16 precision, without replacing the global precision format, without modifying the learning algorithm, and with the inference engine, matrix multiplication kernels, and attention computation kernels all remaining unchanged.

The Nature of the Problem: Two Systems, One Set of Weights, Two Sets of Rounding

Large language model reinforcement learning (whether RLHF or derived methods such as GRPO) typically splits inference and training to run in different systems. On the surface, both sides use the same weights and BF16 tensors, but subtle differences in the underlying implementations can quietly turn gradient updates that should be "on-policy" into "off-policy."

BF16 has a sufficiently large dynamic range, but its mantissa has only 7 bits, far fewer than FP32's 23 bits. When the two systems round the same value at different times, tiny numerical differences are amplified during gradient backpropagation, undermining numerical consistency between training and inference and ultimately triggering policy collapse.

The Cost of Existing Solutions

Previously, the industry mainly had three ways to address this problem: first, align the complete execution stacks of inference and training; second, switch precision globally; third, modify the learning algorithm to introduce additional correction terms.

In October 2025, another research team published a paper on arXiv (arXiv:2510.26788) proposing to change BF16 directly to FP16 with only a few lines of code. That approach held that BF16's excessively large rounding error was the root cause and that uniformly using FP16 could deliver a more stable optimization process and faster convergence, but FP16 has a smaller dynamic range than BF16 and carries overflow risk in some training scenarios.

Surgical Fix: Three Nodes

The new paper by Ke et al. identifies "which computational locations have rounding errors that most affect training stability" and aligns only those locations.

The research team targeted three areas:

• RoPE (Rotary Position Embedding): the computation node for positional information; BF16 precision loss directly affects the consistency of position perception between inference and training;
• SwiGLU: the gated activation function widely used in contemporary large language models, whose multiplicative gating mechanism is sensitive to numerical precision;
• Residual RMSNorm: the normalization layer, a classic accumulation point for cross-layer numerical error.

The fix keeps some intermediate variables temporarily in FP32 during computation in the above three areas and rounds only when finally outputting to BF16. The inference engine, matrix multiplication kernels, attention computation kernels, and learning objective all remain unchanged.

Validation Results and an Unexpected Finding

The paper tested on four models in the Qwen family. Aligning all three areas simultaneously completely prevents the training collapse that occurs under the default BF16 baseline. Aligning the RoPE path alone was also sufficient in all test runs, even though numerical differences in the SwiGLU and RMSNorm areas remained. This points to a concrete engineering conclusion: RoPE is the main source of numerical drift between inference and training.

In November 2024, research from Shanghai AI Laboratory and other institutions revealed that BF16 precision limitations break RoPE relative position encoding in long-context training, and on that basis developed compensation schemes such as AnchorAttention, which in some scenarios shortened training time by more than 50%.

Engineering Value: A Diagnostic Framework

For teams deploying LLM reinforcement learning training pipelines, this paper provides a concrete fault-localization path. In the past, when RL training was unstable, engineers often had only three tactics to try in turn: "switch precision, tune hyperparameters, modify the algorithm." Now there is a precise diagnostic entry point: "check whether the rounding locations of the inference and training stacks are aligned."

The additional memory overhead of keeping FP32 intermediate variables in three local areas is quite limited, far below the memory pressure brought by switching precision globally. Compared with the FP16 approach, this method preserves BF16's advantage of a large dynamic range, which has practical significance for training scenarios with a wide span of gradient values.

Independent Judgment

Switching globally to FP16 is also feasible and simpler to implement in engineering; if a team's training stack allows uniform precision, FP16 is a reasonable choice. The unique contribution of this paper's method is a way of thinking: the root of numerical instability often lies not in algorithmic logic, but in engineering details of operation order and rounding timing.

Validation on four Qwen models is a solid starting point, but it is limited to the same model family. Whether this conclusion can transfer to Llama, Mistral, or other activation function combinations currently has no direct evidence.

From a broader perspective, BF16 training stability fixes are currently advancing along multiple paths—global precision switching, local rounding alignment, and algorithm correction all coexist, with no unified consensus. This shows that the underlying engineering of LLM reinforcement learning has not yet been standardized.