[Survey] Recent approaches on Efficient ML

A collection of recent approaches and papers about Efficient ML including Parameter Efficient Fine Tuning (PEFT), quantization, pruning and other topics.

PEFT

• MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning
• LayerNorm: A key component in parameter-efficient fine-tuning
• ReFT: Representation Finetuning for Language Models
• LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
• LoraPrune: Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning
• LoRA+: Efficient Low Rank Adaptation of Large Models

Quantization

• Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models
• QLoRA: Efficient Finetuning of Quantized LLMs

Pruning

• Random Search as a Baseline for Sparse Neural Network Architecture Search
• The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models