Yuxuan Zhu (UIUC), Tengjun Jin (UIUC), Yoojin Choi (UIUC), and Daniel Kang (Bridgewater AIA Labs and UIUC) in collaboration with Thinking Machines
Many industries rely on relational databases that are queried with SQL. Most SQL is machine-written, but humans alone likely write billions of custom SQL queries each month1 prompted by business questions. They are quite good at it — humans score 92.96% on BIRD, a realistic benchmark for translating natural-language questions into SQL.
However, AI performance on text-to-SQL has lagged behind. LLM scores on the BIRD leaderboard improved from just below 70% in 2024 to 82% today. Frontier models like GPT-5.6 Sol Ultra and Claude Fable 5 can score in the mid-80s, albeit at a cost that is prohibitive for high-volume applications. This isn’t for lack of training data: SQL is widely represented in the internet content used in LLM pretraining. The challenge for AI is in navigating the ambiguous questions and highly-contextual schema that characterize real-world examples.
A common approach for improving AI performance on tasks people understand well is building agentic scaffolding. Systems such as OpenHands, AI co-scientist, and MetaGPT decompose a task into stages, each with its own prompt or model call. Text-to-SQL scaffolds follow the same pattern. A schema-linking stage narrows thousands of columns down to a candidate set.2 A generation stage samples queries. A self-correction stage repairs execution errors. A selection stage votes among the survivors. Every component is a separate call, and the orchestration is typically tuned for a given benchmark.
Scaffolding is an attempt to get over the limitations of model reasoning by making it adhere to a sequence of steps that mirrors how a human would approach the task. And yet, the best scaffolded models still lag 11 points behind humans on SQL. Human professionals acquire their skills with repeated experience, not by being handed a list of instructions — the same should be true of LLMs. Experience with the task should be used to train better reasoning about queries and databases into the model, instead of simply updating the prompts it receives from the scaffold.
In this blog post we describe fine-tuning a model that achieves human-level accuracy on text-to-SQL without scaffolding, using reinforcement learning with verifiable rewards (RLVR) on Tinker.
Extracting the correct answer with SQL is a verifiable task that can be trained in a straightforward manner. However, the performance gap of SQL models suggested that the standard recipe can be improved. Our approach has two crucial improvements: an expert-verified training set purged of the label errors that could poison RLVR, and a reward-shaping technique that targets two common failure modes of RLVR in this domain.
The trained model, ReViSQL-K2.6, exceeds the human mark of 92.96% when picking from 16 samples (SC-16)3 at a cost of $0.56 per task. It is more accurate than Fable 5 and GPT-5.6 Sol Ultra at 12–15% of their cost, and much more accurate than any scaffolded model on the leaderboard.
Code, data, and training recipes are at github.com/uiuc-kang-lab/ReViSQL. We further describe our methods in detail in our technical report.
Curating High-Quality Training Data
RLVR is effective at improving domain-specific reasoning, but it is sensitive to data with incorrect labels. In RLVR, the scalar reward is the entire learning signal for the training step. Mislabeled instances reverse the signal, degrading learning significantly. Our research found that algorithmic tweaks cannot compensate for this loss — cleaning up the data is crucial for RLVR to work well.
We found existing text-to-SQL data to be extremely noisy.4 We sampled 2.5k instances from BIRD Train, a training dataset for text-to-SQL. Our audit found errors in every component of the dataset: the questions, the external knowledge supplied, and in more than half of the “golden SQL queries” that the model’s answer is compared against.
We cleaned up the training set in a multi-stage process. First, an LLM (OpenAI’s o3) and a human expert reviewed each instance and flagged errors. The expert review found that the LLM auditor was precise in catching annotation mistakes (90.6% precision) but only caught 24.5% of the errors flagged by humans. The errors and proposed fixes from this first stage were sent to a different expert for verification. Where the verifier disagreed with the initial auditor, the sample was sent back for additional loops of conflict resolution.
We released the cleaned-up training set to the community as BIRD-Platinum.
We suspected that the evaluation dataset, BIRD Mini-Dev, likewise contained annotation errors. A first cleanup pass of BIRD Mini-Dev was done by Arcwise and corrected errors in 32.3% of instances. We did a second pass that confirmed the vast majority of Arcwise’s flags and found many more, bringing the total detected error rate in BIRD Mini-Dev to 52.8%. The evaluation set with the gold query errors cleaned up was released as Arcwise-Plat-SQL.
BIRD-Platinum Lifts RLVR Above Prior Best Models
We fine-tuned Kimi-K2.6 with RLVR on BIRD-Platinum to produce ReViSQL-K2.6. Training on verified data alone lifted ReViSQL-K2.6 well above both frontier generalist LLMs and the leading open-weight fine-tuned text-to-SQL models on Arcwise-Plat-SQL, with an accuracy score of 88.55%. This shows that the annotation errors in standard training data were the binding constraint on RLVR for text-to-SQL.
To demonstrate that this approach generalizes to other models and eval sets we didn’t touch, we fine-tuned Qwen3-235B-A22B with RLVR on BIRD-Platinum and the original BIRD Train. We tested the model on two new text-to-SQL benchmarks, widely considered more difficult than BIRD:
- Spider2-SQLite: a variant of the Spider2 benchmark that contains complex queries with 5.2× as many tokens on average as Arcwise-Plat-SQL.
- Spider2-Snow: a variant of Spider2 that uses the Snowflake SQL dialect.
Training on the more carefully curated BIRD-Platinum improves the model’s accuracy by 16% on Arcwise-Plat-SQL, 12% on Spider2-SQLite, and 14% on Spider2-Snow compared to BIRD Train. This indicates that our verified data produces a more transferable learning signal across benchmarks and SQL dialects.
Accurate Reward Signal for Text-to-SQL RLVR
Training on clean data brought the fine-tuned model closer to human parity, but a gap of over 4% remained. We looked at the cases where the model failed to identify patterns, which led us to examine the reward function used in training.
Standard text-to-SQL RLVR assigns a reward of 1 when the generated query returns the same result as the gold query on the benchmark database. This mirrors the scoring used in evaluation, but doesn’t fully capture the general behavior we want the model to learn. We focused on two ways result-based reward can diverge from the intended behavior and amended the reward function to address them.
Divergence 1: execution match is not semantic equivalence
The standard result-based reward checks the generated query’s output on a single database instance, but this doesn’t guarantee that the result would hold for a different one. A wrong join key or a dropped predicate can be missed when the particular database instance does not expose the error. Only queries that are semantically equivalent are guaranteed to produce the same result on any database instance.
We test for the semantic equivalence of SQL queries with VeriEQL, a solver that uses bounded verification and incurs negligible CPU costs relative to the total training costs (less than 0.1%). In a pilot training run, we found that 32.8% of positive result-based rewards were given to queries that weren’t fully equivalent to the correct one. That means that nearly one time in three, the reward reinforced the wrong query.
We updated the reward signal by downweighting it in cases where the query was accepted by execution matching but wasn’t equivalent according to VeriEQL. With an additional source of verification, the reward pushes training towards the correct query that generalizes to different database instances.
Divergence 2: outcome rewards are blind to provided knowledge
BIRD-style problems provide external knowledge alongside the question in a prompt. A result-based reward conditions only on the final result, which means it cannot distinguish a model that read the provided information from one that guessed correctly based on its pretraining priors. For example, because “sodium = 0” and “sodium < 5” produce the same result set, a result-based reward cannot distinguish a model that correctly uses external knowledge to choose “sodium = 0” from one that memorizes or hallucinates “sodium < 5.” In the absence of a gradient pushing models to incorporate external knowledge, they tend to default to the prior. In a pilot analysis on a validation set, 24.2% of failures were traced to the model ignoring the necessary information that was supplied.
We address this with rule-based process rewards. The model must emit a requirement block that translates each external-knowledge entry into an explicit query constraint, and a verification block that audits the generated query against those constraints, with penalties for non-compliance. Our process reward encourages the model to ground its reasoning and generation in the provided knowledge rather than blindly relying on its pre-trained prior. The rewards are rule-based rather than model-graded, making the scoring cheap and free from contamination by a judge model’s priors.
Training Recipe
We provide our training recipe for ReViSQL-K2.6. This recipe can also be reproduced using our code built on the Tinker APIs.
Results
We present ReViSQL-K2.6, the model fine-tuned on Tinker with verified data and both reward modifications. Under greedy decoding (single sample, temperature = 0) our model achieves an accuracy of 91.37% on Arcwise-Plat-SQL at a cost of $0.035 per task. This is an 8.4 point improvement over OpenSearch, the strongest prior open-source pipeline, at a 37% lower cost. The cost advantage is a direct consequence of removing auxiliary scaffolding around the model.
If ReViSQL-K2.6 votes among 16 candidates generated with temperature = 1, accuracy rises to 92.97% at a cost of $0.56 per task. This is the first time a text-to-SQL AI system has exceeded the human benchmark to our knowledge.
Conclusion
When AI shows poor performance in a domain-specific task, the common response is to add scaffolding around the task execution. This approach improved performance for text-to-SQL models somewhat, but it ultimately hits the ceiling imposed by the base model’s capability. This doesn’t mean that scaffolding is worthless. Rather, it suggests that the task knowledge contained in the scaffold belongs in the training signal, the same training that would uplift the model’s overall capacity.
The common thread with our previous work on AI trained for financial judgment is that custom models can outperform the frontier on a wide range of tasks involving expert taste and judgment, often at a fraction of the cost. This requires expert judgment to be part of the training process: in identifying how AI fails, labeling the training data, and aligning training to the intended behavior.
In the case of text-to-SQL we saw major improvements from both a rigorous cleanup of the data and the shaping of the reward function to teach the model the correct skill. This took more effort upfront, but the resulting model achieves better performance at lower cost than both humans and scaffolded models. Putting task expertise into task-specific training is ultimately what scales.
1 Based on our internal estimates and publicly available data, such as from Snowflake filings.
2 Real-world, enterprise data systems (including databases, data warehouses, and data lakehouses) contain up to millions of columns. Answering business questions often involves understanding which columns to use. Academic benchmarks are simpler.
3 SC refers to self-consistency selection, where we group concurrently generated SQL queries by their execution results and randomly select a query from the majority group. Neither component is part of a traditional agentic scaffold: sample generation simply draws multiple outputs from the same model prompt, without separately prompted intermediate steps, while majority voting requires no additional model calls.
4 The noisiness of public SQL datasets was noted by many others such as Pourreza and Rafiei (2023) and Wretblad et al. (2024). Our analysis has shown that many widely known benchmarks contain large amounts of noise.
Citation
Please cite this work as:
Zhu, Yuxuan et al., "Human-Level Text-to-SQL via Reinforcement Learning on Verified Data, Without Pipeline Engineering", arXiv:2603.20004 (2026).
Or use the BibTeX citation:
@article{zhu2026revisql,
title = {Human-Level Text-to-SQL via Reinforcement Learning on Verified Data, Without Pipeline Engineering},
author = {Zhu, Yuxuan and Jin, Tengjun and Choi, Yoojin and Kang, Daniel},
journal = {arXiv preprint arXiv:2603.20004},
year = {2026},
note = {https://thinkingmachines.ai/news/putting-task-expertise-into-rl/}
This research paper is prepared by and is the property of Bridgewater Associates, LP and is circulated for informational and educational purposes only. There is no consideration given to the specific investment needs, objectives, or tolerances of any of the recipients. Additionally, Bridgewater’s actual investment positions may, and often will, vary from its conclusions discussed herein based on any number of factors, such as client investment restrictions, portfolio rebalancing and transactions costs, among others. Recipients should consult their own advisors, including tax advisors, before making any investment decision. This material is for informational and educational purposes only and is not an offer to sell or the solicitation of an offer to buy the securities or other instruments mentioned. Any such offering will be made pursuant to a definitive offering memorandum. This material does not constitute a personal recommendation or take into account the particular investment objectives, financial situations, or needs of individual investors which are necessary considerations before making any investment decision. Investors should consider whether any advice or recommendation in this research is suitable for their particular circumstances and, where appropriate, seek professional advice, including legal, tax, accounting, investment, or other advice. No discussion with respect to specific companies should be considered a recommendation to purchase or sell any particular investment. The companies discussed should not be taken to represent holdings in any Bridgewater strategy. It should not be assumed that any of the companies discussed were or will be profitable, or that recommendations made in the future will be profitable.
The information provided herein is not intended to provide a sufficient basis on which to make an investment decision and investment decisions should not be based on simulated, hypothetical, or illustrative information that have inherent limitations. Unlike an actual performance record simulated or hypothetical results do not represent actual trading or the actual costs of management and may have under or overcompensated for the impact of certain market risk factors. Bridgewater makes no representation that any account will or is likely to achieve returns similar to those shown. The price and value of the investments referred to in this research and the income therefrom may fluctuate. Every investment involves risk and in volatile or uncertain market conditions, significant variations in the value or return on that investment may occur. Investments in hedge funds are complex, speculative and carry a high degree of risk, including the risk of a complete loss of an investor’s entire investment. Past performance is not a guide to future performance, future returns are not guaranteed, and a complete loss of original capital may occur. Certain transactions, including those involving leverage, futures, options, and other derivatives, give rise to substantial risk and are not suitable for all investors. Fluctuations in exchange rates could have material adverse effects on the value or price of, or income derived from, certain investments.
Bridgewater research utilizes data and information from public, private, and internal sources, including data from actual Bridgewater trades. Sources include AERIC INC, BCA, Bloomberg Finance L.P., Candeal, Carbon Arc, CEIC Data Company Ltd., Ceras Analytics, China Bull Research, Citibank, Clarus Financial Technology, CLS Processing Solutions, Consensus Economics Inc., Consumer Edge, CRU Group, DTCC Data Repository, Ecoanalitica, Energy Aspects Corp, Enverus, EPFR Global, Eurasia Group, Evercore ISI, FactSet Research Systems, The Financial Times Limited, Finaeon, Inc., FINRA, GaveKal Research Ltd., GlobalSource Partners, Goldman Sachs, Harvard Business Review, Haver Analytics, Inc., IEA, Institutional Shareholder Services (ISS), The Investment Funds Institute of Canada, ICE Derived Data (UK), Investment Company Institute, International Institute of Finance, JP Morgan, JTSA Advisors, LSEG Data and Analytics, MarketAxess, Metals Focus Ltd, MSCI, Inc., National Bureau of Economic Research, Neudata, Organisation for Economic Cooperation and Development, Pensions & Investments Research Center, Pitchbook, Political Alpha, Renaissance Capital Research, Rhodium Group, RP Data, Rubinson Research, Rystad Energy, S&P Global Market Intelligence, Sentix GmbH, SGH Macro, Shanghai Metals Market, Smart Insider Ltd., Swaps Monitor, Tradeweb, United Nations, US Department of Commerce, Visible Alpha, Wells Bay, Wind Financial Information LLC, With Intelligence, Wood Mackenzie Limited, World Bureau of Metal Statistics, World Economic Forum, and YieldBook. While we consider information from external sources to be reliable, we do not assume responsibility for its accuracy. Data leveraged from third-party providers, related to financial and non-financial characteristics, may not be accurate or complete. The data and factors that Bridgewater considers within its research process may change over time.
This information is not directed at or intended for distribution to or use by any person or entity located in any jurisdiction where such distribution, publication, availability, or use would be contrary to applicable law or regulation, or which would subject Bridgewater to any registration or licensing requirements within such jurisdiction. No part of this material may be (i) copied, photocopied, or duplicated in any form by any means or (ii) redistributed without the prior written consent of Bridgewater® Associates, LP.
The views expressed herein are solely those of Bridgewater as of the date of this report and are subject to change without notice. Bridgewater may have a significant financial interest in one or more of the positions and/or securities or derivatives discussed. Those responsible for preparing this report receive compensation based upon various factors, including, among other things, the quality of their work and firm revenues.