Singular Value Representation: A Graph Perspective on Neural Networks

Created in July 09, 2023

2023   ·   theory   ·   paper-review

This paper develops a graph-theoretic perspective on neural networks built on the singular value decomposition (SVD) of weight matrices.

The core idea: each layer’s weight matrix admits an SVD, and the right and left singular vectors form a graph connecting the input and output bases. Properties of this graph (sparsity, connectivity, spectral structure) reflect what the layer does in terms of mixing input dimensions — providing a more interpretable lens than treating weights as a black box.

This perspective enables:

  • Diagnosing redundancy via low-rank structure
  • Tracking how representations transform across layers
  • Connecting initialization, training dynamics, and final structure through spectral evolution

The framework offers a principled tool for analyzing trained networks rather than just measuring their performance.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition