Reinforcement learning with verifiable rewards (RLVR) has become a critical tool for modern large language model (LLM) development. RLVR is used to improve the general-purpose reasoning abilities of frontier language models, and has proven particularly critical for mathematical problem solving and agentic coding. Beyond the frontier, a model which is trained via RLVR can outperform much larger frontier models in specialized domains, making this approach especially attractive for enterprise tasks involving proprietary data. For example, Databricks uses RLVR to train a text-to-SQL specialist that outperforms much larger frontier models in this domain.
Unfortunately, this kind of specialization often comes at the risk of overfitting: it can be unclear whether a model trained via RLVR has learned to generalize in deployment, or if it is simply memorizing its training data. Empirical studies have reported mixed results [1, 2, 3], and our theoretical understanding of generalization remains limited in this context. Can we move beyond empirics to theoretically certify the generalizability of models trained via RLVR?
In this blog post, we introduce a framework that verifies the generalizability of RLVR models on fresh data drawn from the same distribution as the training data, and establish the first non-vacuous generalization bounds for RLVR at the billion-parameter scale. Across four real-world domains (math, programming, general-knowledge reasoning, and Text-to-SQL), our lower bounds for Qwen3.5-4B exceed the base model’s accuracy by 9–51%.
For more details, please check out our paper and code.
PAC-Bayes Compression Bounds for RLVR
TL;DR. We adopt the PAC-Bayes framework for proving generalization bounds. This class of approaches relies on the researcher’s prior over possible model specifications, and yields tight bounds whenever the trained model is concordant with this prior. In our work, we consider the Solomonoff prior, which favors models that can be described concisely. Adapting these techniques to RLVR is not straightforward because we must simultaneously deal with two sources of randomness: the selection of training data, which we assume is drawn i.i.d. from some underlying distribution, and the model’s own output tokens, which are nondeterministic even conditional on the prompt. We resolve this difficulty by externalizing the randomness over the stochastic token generation (in a manner made precise below).
For this mathematical guarantee to be meaningful for a billion-parameter model trained on a few hundred thousand prompts, we must compress the trained model into a few kilobytes. Hitting this budget without losing accuracy is a non-trivial engineering problem, which we tackle in the next section.
(More interested in the practical pipeline than the theoretical derivation? Skip to the next section.)
—
Classic generalization bounds that depend on the complexity of the model class under consideration (e.g., VC dimension) become vacuous at the billion-parameter scale. PAC-Bayes bounds take a different approach by evaluating the trained model against a prior over possible model specifications. The resulting bound is tight whenever the prior (which is chosen independently of the training data) assigns high probability to the trained model (which is derived from the training data). In this work we adopt the Solomonoff prior, which favors models that can be described concisely; this can be viewed as an application of the Occam’s-razor principle. Specifically, we begin with the following compression-based bound, which measures the generalization gap via the size of the model’s compressed representation.
While the bound above applies to pre-training with the Negative Log-Likelihood (NLL) loss, it becomes intractable for RLVR. Unlike the NLL loss, which computes a deterministic number for an input prompt, the core objective of RLVR is defined over stochastically decoded tokens:
The inner expectation calculates a rule-based reward over the entire sequence generation distribution, making it intractable. To tackle this, we apply the Gumbel-max trick, which turns a stochastic process (drawing from a categorical distribution) into a deterministic function of random noise:
We then model the stochastic token decoding process as a deterministic function of an input prompt, Gumbel noise, and a hypothesis, which in turn results in an approximation of the training objective:
The final step is to apply PAC-Bayes bounds twice to measure:
- Intrinsic penalty: the gap between the approximated training objective and the true training objective, and
- External penalty: the gap between the training objective and the population objective.
Putting them together yields the final bound:
This bound is non-vacuous only when C(h) is small with respect to m, requiring us to compress the trained model to far fewer bits than the number of training prompts. Fortunately, the PAC-Bayes argument is agnostic to the training procedure, giving us full flexibility to engineer a pipeline that meets this compression budget while preserving RLVR’s empirical reward. We describe such a pipeline in the next section.
Achieving a High Compression Ratio via Progressive RLVR
Before we dive into our pipeline, let’s first understand how the standard approach fails. Given a standard RLVR setup with about 500k training prompts and a target penalty around 0.1, we need C(h) on the order of a few kilobytes, which is more than 500 times smaller than a minimal standard LoRA adapter (rank=1) on a 4B base model.
While simultaneously achieving small compression size and large rewards is challenging, one advantage of the PAC-Bayes framework is that it does not imply any constraints on the training procedure. This decouples capacity used during training from capacity charged to the bound. Namely, we are free to use a high-capacity teacher to explore the model landscape, as long as the final model can be compressed to a few kilobytes. To exploit this asymmetry, we introduce the following Progressive RLVR pipeline:
- On-policy distillation: Running RLVR with a small parameter budget often leads to sub-optimal results because the reward signal is sparse. Instead, we first train a teacher without a parameter budget, and then distill it into a small student via on-policy distillation, which provides dense, token-level supervision.
- TinyLoRA: Standard LoRA parameterization requires at least d (input dimension) + k (output dimension) per layer, which is still too many parameters to achieve non-vacuous bounds. In contrast, TinyLoRA reparameterizes the weight updates into an arbitrary number of trainable parameters, leveraging truncated SVD and randomized projection:
- Quantization: After training, we quantize the TinyLoRA adapter into a small number of discrete, uniform levels, which significantly reduces the compression size with mild accuracy degradation.
Non-vacuous Bounds for Billion-Parameter LLMs
We evaluate the full pipeline on Qwen3.5-4B across four real-world RLVR domains:
- Mathematical problem-solving: Eurus-2-RL-Math (456k prompts),
- Programming: Eurus-2-RL-Code and KodCode-V1 (505k prompts)
- General-knowledge reasoning: Med-QA, MMLU, LegalBench, LogiQA, C-Eval, and ARC (439k prompts)
- Text-to-SQL: SynSQL-2.5M (2.5M prompts).
For mathematical problem-solving, programming, and general-knowledge reasoning, we use binary rewards {0, 1}, which directly reflect accuracy. For Text-to-SQL, we use a tri-level reward structure {-1, 0, 1} to penalize unparsable responses. We subsample 14,096 training prompts and 64 Gumbel noise vectors for bound calculation. We use a failure probability of 0.05 and a quantization level of 5 (about 2.3 bits) by default.
As shown in the figure above, our bound exceeds the empirical accuracy of the base model and tracks the student’s empirical accuracy closely. Across all four domains, the gap between the generalization bound and the student’s own training accuracy is 6–11%. Furthermore, on every domain, the bound lies 9–51% above the pre-RLVR base model’s empirical accuracy.
Digging deeper, we decompose the bounds into specific penalties: intrinsic penalty, external penalty, distillation-and-quantization penalty, and subsampling penalty. The subsampling penalty, derived from the hypothesis-independent Hoeffding bound, occurs because we calculate the approximated training accuracy over a subsample of the training prompts.
As shown in the figure above, we find that the intrinsic and subsampling penalties are small across all four domains, confirming that our subsample size and number of noise draws are sufficient. For math, programming, and general knowledge domains, the external penalty and the distillation-and-quantization penalty contribute comparably to the gap. Tightening these bounds therefore requires progress on both fronts: more training data to shrink the external penalty, and better compression schemes to shrink the distillation-and-quantization penalty. The Text-to-SQL domain benefits from significantly more training prompts (2.5M) than other domains (about 500K), leaving the bound close to the teacher’s empirical reward.
Finally, we perform an ablation study to understand the importance of each ingredient of Progressive RLVR. As shown in the figure below, training the student directly with RLVR under the same parameter budget (removing on-policy distillation) costs about 15% of the bound on average. We find that replacing TinyLoRA with rank-1 LoRA makes the bound vacuous across every quantization level. Even when we shrink the base to Qwen3.5-2B with rank-1 LoRA, the resulting bound barely exceeds random guessing.
As illustrated in the plot, we sweep over various quantization levels and identify a non-monotonic trade-off between the bound and the quantization level. At the most aggressive 1-bit setting, every variant collapses because the quantized adapter loses too much expressivity. At the other extreme, adding precision increases C(h) faster than it improves the training accuracy, leading to increasingly loose bounds as the number of levels grows.
Our experimentation was enabled by Thinking Machine’s Tinker, which accelerated our research by providing great infrastructure to experiment with.
Conclusion
RLVR is the engine behind today’s frontier reasoning models, yet its ability to generalize to unseen data has remained a theoretical blind spot. This uncertainty is a major bottleneck for enterprise AI. Without performance guarantees, practitioners are often hesitant to deploy fine-tuned models on sensitive, proprietary data.
By pairing a compression-based PAC-Bayes framework with a highly tailored training pipeline, we establish the first non-vacuous generalization bounds for RLVR. Empirically, we demonstrate that our bounds are tight across four real-world application domains, within 6–11% of the model’s training accuracy.
While our framework certifies in-domain generalization for stationary environments, the next frontier lies in dynamic settings (e.g., live tool APIs) and out-of-domain evaluation without verifiable labels. Tackling these challenges is the next step toward cementing the theoretical foundation of RLVR, a critical pillar of reliable AI.
Find more details about our work here:
Paper: https://arxiv.org/abs/2607.14506
Code: https://github.com/uiuc-kang-lab/rlvr_generalization_bounds
This research paper is prepared by and is the property of Bridgewater Associates, LP and is circulated for informational and educational purposes only. There is no consideration given to the specific investment needs, objectives, or tolerances of any of the recipients. Additionally, Bridgewater’s actual investment positions may, and often will, vary from its conclusions discussed herein based on any number of factors, such as client investment restrictions, portfolio rebalancing and transactions costs, among others. Recipients should consult their own advisors, including tax advisors, before making any investment decision. This material is for informational and educational purposes only and is not an offer to sell or the solicitation of an offer to buy the securities or other instruments mentioned. Any such offering will be made pursuant to a definitive offering memorandum. This material does not constitute a personal recommendation or take into account the particular investment objectives, financial situations, or needs of individual investors which are necessary considerations before making any investment decision. Investors should consider whether any advice or recommendation in this research is suitable for their particular circumstances and, where appropriate, seek professional advice, including legal, tax, accounting, investment, or other advice. No discussion with respect to specific companies should be considered a recommendation to purchase or sell any particular investment. The companies discussed should not be taken to represent holdings in any Bridgewater strategy. It should not be assumed that any of the companies discussed were or will be profitable, or that recommendations made in the future will be profitable.
The information provided herein is not intended to provide a sufficient basis on which to make an investment decision and investment decisions should not be based on simulated, hypothetical, or illustrative information that have inherent limitations. Unlike an actual performance record simulated or hypothetical results do not represent actual trading or the actual costs of management and may have under or overcompensated for the impact of certain market risk factors. Bridgewater makes no representation that any account will or is likely to achieve returns similar to those shown. The price and value of the investments referred to in this research and the income therefrom may fluctuate. Every investment involves risk and in volatile or uncertain market conditions, significant variations in the value or return on that investment may occur. Investments in hedge funds are complex, speculative and carry a high degree of risk, including the risk of a complete loss of an investor’s entire investment. Past performance is not a guide to future performance, future returns are not guaranteed, and a complete loss of original capital may occur. Certain transactions, including those involving leverage, futures, options, and other derivatives, give rise to substantial risk and are not suitable for all investors. Fluctuations in exchange rates could have material adverse effects on the value or price of, or income derived from, certain investments.
Bridgewater research utilizes data and information from public, private, and internal sources, including data from actual Bridgewater trades. Sources include AERIC INC, BCA, Bloomberg Finance L.P., Candeal, Carbon Arc, CEIC Data Company Ltd., Ceras Analytics, China Bull Research, Citibank, Clarus Financial Technology, CLS Processing Solutions, Consensus Economics Inc., Consumer Edge, CRU Group, DTCC Data Repository, Ecoanalitica, Energy Aspects Corp, Enverus, EPFR Global, Eurasia Group, Evercore ISI, FactSet Research Systems, The Financial Times Limited, Finaeon, Inc., FINRA, GaveKal Research Ltd., GlobalSource Partners, Goldman Sachs, Harvard Business Review, Haver Analytics, Inc., IEA, Institutional Shareholder Services (ISS), The Investment Funds Institute of Canada, ICE Derived Data (UK), Investment Company Institute, International Institute of Finance, JP Morgan, JTSA Advisors, LSEG Data and Analytics, MarketAxess, Metals Focus Ltd, MSCI, Inc., National Bureau of Economic Research, Neudata, Organisation for Economic Cooperation and Development, Pensions & Investments Research Center, Pitchbook, Political Alpha, Renaissance Capital Research, Rhodium Group, RP Data, Rubinson Research, Rystad Energy, S&P Global Market Intelligence, Sentix GmbH, SGH Macro, Shanghai Metals Market, Smart Insider Ltd., Swaps Monitor, Tradeweb, United Nations, US Department of Commerce, Visible Alpha, Wells Bay, Wind Financial Information LLC, With Intelligence, Wood Mackenzie Limited, World Bureau of Metal Statistics, World Economic Forum, and YieldBook. While we consider information from external sources to be reliable, we do not assume responsibility for its accuracy. Data leveraged from third-party providers, related to financial and non-financial characteristics, may not be accurate or complete. The data and factors that Bridgewater considers within its research process may change over time.
This information is not directed at or intended for distribution to or use by any person or entity located in any jurisdiction where such distribution, publication, availability, or use would be contrary to applicable law or regulation, or which would subject Bridgewater to any registration or licensing requirements within such jurisdiction. No part of this material may be (i) copied, photocopied, or duplicated in any form by any means or (ii) redistributed without the prior written consent of Bridgewater® Associates, LP.
The views expressed herein are solely those of Bridgewater as of the date of this report and are subject to change without notice. Bridgewater may have a significant financial interest in one or more of the positions and/or securities or derivatives discussed. Those responsible for preparing this report receive compensation based upon various factors, including, among other things, the quality of their work and firm revenues.