[Survey] Recent LLM Technical Reports

A collection of recent technical reports from several vendors including Google DeepMind, xAI, AllenAI, AI21Labs, Databricks, and HyperCLOVA.

Most reports cover extensive evaluation of foundation LLMs and observations such as scaling laws.

Foundation LLMs

• HyperCLOVA X Technical Report
• Stable Code Technical Report
• [AI21Labs] Jamba: A Hybrid Transformer-Mamba Language Model
• [DeepMind] Gecko: Versatile Text Embeddings Distilled from Large Language Models
• [xAI] Grok-1.5
• [Databricks] DBRX: Introducing a New State-of-the-Art Open LLM
• [DeepMind] Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens

Evaluation

• [DeepMind] Evaluating Frontier Models for Dangerous Capabilities
• Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra
• Evaluating LLMs at Detecting Errors in LLM Responses
• [Microsoft] Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning
• [AllenAI] Evaluating Reward Models for Language Modeling
• [DeepMind] Gemma: Open Models Based on Gemini Research and Technology

Scaling Laws

• Unraveling the Mystery of Scaling Laws: Part I
• Scaling Laws For Dense Retrieval
• Scaling Data-Constrained Language Models