Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Created in October 27, 2025

2025   ·   ssm   llm   ·   paper-review

Mamba (Gu & Dao, 2023) introduces a selective state space model that resolves the efficiency gap between SSMs and Transformers:

Key innovation — input-dependent transitions: Standard SSMs have fixed $A, B, C$ matrices. Mamba makes these data-dependent (parameterized by the input sequence), enabling selective filtering of information — the model learns what to remember and what to forget based on content, analogous to an LLM’s attention but without $O(L^2)$ cost.

Hardware-aware algorithm: A parallel scan algorithm exploits GPU memory hierarchy to avoid materializing the expanded state, achieving near-linear compute in practice.

Results:

  • Matches or exceeds Transformer quality on language modeling at equivalent parameter counts
  • Inference scales linearly with sequence length (vs. quadratic for attention)
  • Particularly strong on long-sequence tasks where Transformers saturate

Paper: arXiv:2312.00752



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition