HICRA: Hierarchical Credit Assignment for LLM Reasoning

Key Finding

HICRA identifies that reinforcement learning improves LLM reasoning through an emergent two-phase hierarchy:

1. Early training: fixes low-level procedural tokens (arithmetic steps, formatting)
2. Later training: gains come from high-level strategic planning tokens (problem decomposition, approach selection)

Standard GRPO distributes credit uniformly across all tokens, missing this hierarchy.

Method

Rather than uniform credit assignment, HICRA amplifies gradients on planning tokens — the actual bottleneck for reasoning improvement. This targeted credit assignment reflects where the model’s decisions actually matter.

Evidence for the hierarchy:

• “Aha moments” cluster around strategic decision points, not procedural execution
• Length scaling patterns in RL training align with planning-token density

Results

• Outperforms GRPO-based approaches on mathematical and multi-step reasoning tasks
• Integrates into various RL frameworks as an incremental modification
• Provides both an empirical observation and an actionable algorithm

Paper: arXiv:2509.03646