Robust Reasoning RL
Beyond a higher reward. Studying reasoning when the judge itself is imperfect.
Do reasoning policies learn correctness, or exploit noisy and inconsistent verifier rewards?
INTERACTIVE WORKFLOW
Robust RL / reward is not correctness
A framework and pilot for imperfect-verifier experiments. The final GPU study is blocked and the primary hypothesis remains unestablished.
Dataset
Use matched prompts, data and budgets for baseline and robust methods.
Matched inputs
Read the complete workflow
- Dataset
Use matched prompts, data and budgets for baseline and robust methods.
Framework + pilot · Output: Matched inputs - Corruption generation
Controlled verifier corruption models noisy, incomplete, biased and inconsistent rewards.
Documented method · Output: Corruption configuration - Policy model
Generate candidate reasoning responses with the current policy and record each generation.
Pretrained pilot completed · Output: Candidate responses - Training / optimization
Apply uncertainty-aware reward shaping on a shared GRPO + LoRA training path. Reward shaping does not guarantee correctness.
Pilot / final GPU study blocked · Output: Updated policy - Verification
Evaluate independent correctness rather than treating training reward as capability.
Pilot evidence · Output: Independent scores - Robustness evaluation
Compare corruption sweeps and reward gaps. The full GPU study remains blocked; the primary hypothesis is unestablished.
Final study pending · Output: Qualified findings
What the study tests
Uncertainty-aware reward shaping might reduce the gap between corrupted verifier reward and independently judged correctness relative to standard GRPO.
Methodology and algorithms
Shared baseline and robust GRPO paths compare reward shaping under controlled verifier corruption. Per-generation records separate observed reward from independent correctness.
Controls and comparisons
Match prompts, generation budgets and training settings across methods. Use independent evaluation rather than treating the optimization reward as proof of capability.
What the results say
The repository reports a completed framework and pilot, including pretrained save/reload runs. The primary research hypothesis remains unestablished; the final GPU study is externally blocked.
Where the evidence ends
Software fixtures are not RL experimental evidence. Pilot runs do not establish statistical superiority or generalization.
Tests and validation
The 27-condition SmolLM2 CPU pilot completed but produced zero independently correct answers. Qwen qualification and one-update CPU smoke are separate evidence. The final GPU comparison has not run.
Reproduce the investigation
Use the final-report links to the pilot aggregates, original generations, Qwen qualification and final execution package. Keep the four-question pilot separate from later qualification and smoke results; the final study requires CUDA.
Source reviewed 17 October 2026. Summaries describe repository evidence; they do not assert sole authorship or independent reproduction.
Read the source documentation ↗Credit belongs to the repository’s contributors. Account or organization ownership does not establish sole authorship; individual contribution boundaries have not been verified.