SOAR — Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

Paper. Shobhita Sundaram, John Quan, Ariel Kwiatkowski, Kartik Ahuja, Yann Ollivier, Julia Kempe. Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability. arXiv:2601.18778, Feb 2026. MIT · NYU CDS · Meta FAIR. [arXiv] · [project page]

0. The Picture in One Paragraph

Reinforcement learning has become the standard recipe for pushing LLM reasoning skills. But RL has a hard wall: if the base model never solves a problem, RL has no signal to learn from. On the hardest math problems where the base model fails every one of 128 attempts, RL flatlines. SOAR proposes a way past this wall: train a teacher copy of the model whose job is to invent stepping-stone problems for a student copy, and pay the teacher in the currency of the student’s measured improvement on real hard problems. The result is a bilevel meta-RL loop that unlocks learning on benchmarks where base RL gets nothing — 4× pass@1 on MATH, 2× on HARP — using nothing but the model’s own latent curriculum-design capacity.

1. Why Vanilla RL Stalls on Hard Problems

RL for reasoning (RLVR, GRPO, REINFORCE++, …) optimizes a simple objective: roll out the model on a problem, check whether the answer is correct, and update toward whatever produced the correct answer.

There’s a hidden precondition: at least some rollouts must succeed. If the model never solves the problem, every rollout has reward 0, the policy gradient is 0, and RL doesn’t move the parameters. This is the “fail@K” regime — and on competition math (MATH, HARP, OlympiadBench), a non-trivial fraction of problems sit in fail@128 for moderate-size models.

You can try three obvious workarounds, and the paper rules each one out:

SOAR’s argument is that self-play is the right shape — but the teacher’s reward function has been the broken piece.

2. The SOAR Framework

The full name is Self-Optimization via Asymmetric RL. The asymmetry is the key word: teacher and student have different jobs, different action spaces, and different rewards.

2.1 The setup

• Two copies of the same model, both initialized from the target LLM (in the paper, Llama-3.2-3B-Instruct).
• One copy becomes the student. Its job: solve problems by generating chains of thought, then update its policy with RL.
• The other copy becomes the teacher. Its job: generate problems and proposed answers for the student to train on.

2.2 The two loops

Outer loop — teacher learns to teach
  for each iteration:
      teacher generates a batch of (question, answer) pairs   ← teacher policy

      ─── Inner loop — student trains briefly on those problems ───
      run short-burst RL on the student
          (≈ 10–20 update steps, batch size 8)
      ─────────────────────────────────────────────────────────────

      measure student's improvement on a small held-out set of
      *real, ground-truth hard problems*

      use that improvement as the teacher's reward
      update the teacher with RLOO

The student is a throwaway probe in the outer loop: each outer iteration spins up a fresh student RL phase, measures its improvement, then discards it. The teacher accumulates the signal across iterations.

2.3 The reward design — the contribution

Prior self-play methods reward the teacher with intrinsic proxies — “is this problem novel?”, “is it hard?”, “is the answer correct?”. These proxies have no necessary connection to whether the student can actually learn from the problem.

SOAR replaces every proxy with one grounded signal:

Reward the teacher in proportion to how much the student improved on real hard problems.

This single substitution is what makes the loop work. Whatever the teacher discovers about good problem design has to bottom out in measurable learning progress on the actual benchmark.

3. The Counterintuitive Finding About “Quality”

If the teacher’s job is to produce useful training data, you’d guess the most important feature is correctness — does the proposed answer actually solve the proposed problem? SOAR’s analysis says no:

Structural quality and well-posedness matter more than solution correctness.

A clear, unambiguous, well-scaffolded problem with a wrong answer still produces a learnable gradient for the student. An ambiguous or malformed problem with a correct answer doesn’t.

This matters for two reasons:

1. It explains why bootstrapping is possible. The teacher’s latent ability to produce well-posed questions is much stronger than its ability to solve hard problems. SOAR taps the former.
2. It rewrites the conventional wisdom that synthetic data must be filtered for solution correctness before training. SOAR works without that filter.

4. Diversity and Stability

Self-play has historically collapsed onto a narrow set of strategies. SOAR measures this directly using the Vendi Score (a semantic diversity metric over the teacher’s generated questions). The finding:

• Grounded teacher (SOAR): diversity comparable to the base model — stable, broad, non-collapsing.
• Intrinsic-reward teacher: severe diversity collapse — the teacher converges to a few stereotyped problems and the loop destabilizes.

The mechanism is intuitive: an intrinsic reward will always be maximized by Goodhart-ing the proxy (find the cheapest way to look “novel” or “hard”); a grounded reward can only be maximized by genuinely helping the student learn, which requires distributional breadth.

5. Headline Results

All numbers are with Llama-3.2-3B-Instruct, evaluated on the fail@128 subsets of each benchmark (problems where the base model never succeeds in 128 attempts).

These are unlocks, not refinements. The base model is at the literal floor; SOAR drives it off the floor without any external teacher and without any new ground-truth data beyond a small held-out probe set used to score the teacher.

6. What the Method Actually Says

The structural claim of the paper, stripped of acronyms:

A pretrained LLM already possesses the latent capacity to design a useful curriculum for itself. Standard RL cannot access this capacity because the model’s solving ability and its curriculum-design ability are not the same skill. A bilevel loop where the latter is rewarded by the former exposes the latent capacity and turns it into measurable learning.

This recasts a familiar question — “how do we get a model past a plateau?” — into a different one: “what part of the model is already above the plateau, and how do we route reward to it?”

7. Limitations Worth Knowing

• Scale. Results are at 3B parameters. Whether the curriculum-design capacity scales with parameters (and whether SOAR is needed at all once base models cross some capability threshold) is an open question.
• Reward latency. The teacher’s reward signal requires running a fresh student RL phase — outer-loop steps are expensive.
• Probe set dependency. The grounded reward is computed on a small subset of real hard problems. If that subset is unrepresentative, the teacher will optimize for the wrong thing.
• Modality. The paper covers math. Whether structural well-posedness is the right axis for code, science, or open-ended-reasoning benchmarks is plausible but unverified.

8. Why This Matters

Two reasons:

1. It targets the most painful wall in reasoning post-training. Most of the gains from RL evaporate on the hardest problems — the ones we most want models to solve. SOAR is the first recipe I’ve seen that produces non-trivial movement on fail@128 problems without external supervision.
2. It re-frames self-improvement as a teacher-design problem. The asymmetry isn’t a hack; it reflects a real fact about pretrained models — generating well-posed problems is a different skill than solving them. Future self-improvement methods that take this seriously have a credible direction to walk in.

For context, this sits very close to the Fast-Slow Training trend post on this site — both papers argue that the way we currently spend RL samples is wasteful, and both propose loops that extract more signal per rollout. SOAR’s contribution is at the task-distribution level (what does the model train on?); FST’s is at the adaptation-surface level (what does the model update?).

9. TL;DR

If you remember three things:

1. RL stalls when initial success is 0 because zero rewards give zero gradient.
2. SOAR is a bilevel meta-RL loop where a teacher copy of the model generates problems and is rewarded in proportion to the student’s measured improvement on real hard problems — replacing the proxy rewards that destabilize prior self-play.
3. Grounded rewards beat intrinsic ones, structural quality of the generated problems beats solution correctness, and the result is a 4× pass@1 improvement on the MATH fail@128 subset using nothing but the base model’s own latent curriculum capacity.

References

• Sundaram, S., Quan, J., Kwiatkowski, A., Ahuja, K., Ollivier, Y., & Kempe, J. (2026). Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability. arXiv:2601.18778.
• Project page: https://ssundaram21.github.io/soar/
• Related on this site: Fast-Slow Training trend note, Meta-Harness review.