REFRAG: Rethinking RAG-Based Decoding

Problem

In retrieval-augmented generation (RAG), most tokens retrieved from documents are irrelevant to any given decoding step. Processing all retrieved content uniformly is both slow and wasteful.

Method: Compress → Sense → Expand

REFRAG operates in three stages:

1. Compress: Reduce retrieved passages to a compact representation
2. Sense: Identify which passage segments are relevant at each decoding step
3. Expand: Reconstruct only the necessary context for generation

This exploits a key structural property of RAG workflows: block-diagonal attention patterns naturally emerge, meaning tokens within retrieved chunks attend primarily to each other. REFRAG makes this structure explicit to skip irrelevant blocks.

Results

• ~31× speedup in time-to-first-token (TTFT)
• Supports contexts up to 16× longer without accuracy loss
• No architecture changes — works with existing models and requires no retraining

Significance

REFRAG addresses a practical bottleneck for production RAG systems: long retrieval contexts dominate inference latency. By targeting the structural inefficiency rather than the model, the gains apply broadly without model-specific engineering.