MLG 034 Large Language Models 1

Machine Learning Guide

Explains language models (LLMs) advancements. Scaling laws - the relationships among model size, data size, and compute - and how emergent abilities such as in-context learning, multi-step reasoning, and instruction following arise once certain scaling thresholds are crossed. The evolution of the transformer architecture with Mixture of Experts (MoE), describes the three-phase training process culminating in Reinforcement Learning from Human Feedback (RLHF) for model alignment, and explores advanced reasoning techniques such as chain-of-thought prompting which significantly improve complex task performance.

Links

• Notes and resources at ocdevel.com/mlg/mlg34
• Build the future of multi-agent software with AGNTCY
• Try a walking desk stay healthy & sharp while you learn & code

Transformer Foundations and Scaling Laws

• Transformers: Introduced by the 2017 "Attention is All You Need" paper, transformers allow for parallel training and inference of sequences using self-attention, in contrast to the sequential nature of RNNs.
• Scaling Laws:
  • Empirical research revealed that LLM performance improves predictably as model size (parameters), data size (training tokens), and compute are increased together, with diminishing returns if only one variable is scaled disproportionately.
  • The "Chinchilla scaling law" (DeepMind, 2022) established the optimal model/data/compute ratio for efficient model performance: earlier large models like GPT-3 were undertrained relative to their size, whereas right-sized models with more training data (e.g., Chinchilla, LLaMA series) proved more compute and inference efficient.

Emergent Abilities in LLMs

• Emergence: When trained beyond a certain scale, LLMs display abilities not present in smaller models, including:
  • In-Context Learning (ICL): Performing new tasks based solely on prompt examples at inference time.
  • Instruction Following: Executing natural language tasks not seen during training.
  • Multi-Step Reasoning & Chain of Thought (CoT): Solving arithmetic, logic, or symbolic reasoning by generating intermediate reasoning steps.
• Discontinuity & Debate: These abilities appear abruptly in larger models, though recent research suggests that this could result from non-linearities in evaluation metrics rather than innate model properties.

Architectural Evolutions: Mixture of Experts (MoE)

• MoE Layers: Modern LLMs often replace standard feed-forward layers with MoE structures.
  • Composed of many independent "expert" networks specializing in different subdomains or latent structures.
  • A gating network routes tokens to the most relevant experts per input, activating only a subset of parameters—this is called "sparse activation."
  • Enables much larger overall models without proportional increases in compute per inference, but requires the entire model in memory and introduces new challenges like load balancing and communication overhead.
• Specialization & Efficiency: Experts learn different data/knowledge types, boosting model specialization and throughput, though care is needed to avoid overfitting and underutilization of specialists.

The Three-Phase Training Process

• 1. Unsupervised Pre-Training: Next-token prediction on massive datasets—builds a foundation model capturing general language patterns.
• 2. Supervised Fine Tuning (SFT): Training on labeled prompt-response pairs to teach the model how to perform specific tasks (e.g., question answering, summarization, code generation). Overfitting and "catastrophic forgetting" are risks if not carefully managed.
• 3. Reinforcement Learning from Human Feedback (RLHF):
  • Collects human preference data by generating multiple responses to prompts and then having annotators rank them.
  • Builds a reward model (often PPO) based on these rankings, then updates the LLM to maximize alignment with human preferences (helpfulness, harmlessness, truthfulness).
  • Introduces complexity and risk of reward hacking (specification gaming), where the model may exploit the reward system in unanticipated ways.

Advanced Reasoning Techniques

• Prompt Engineering: The art/science of crafting prompts that elicit better model responses, shown to dramatically affect model output quality.
• Chain of Thought (CoT) Prompting: Guides models to elaborate step-by-step reasoning before arriving at final answers—demonstrably improves results on complex tasks.
  • Variants include zero-shot CoT ("let's think step by step"), few-shot CoT with worked examples, self-consistency (voting among multiple reasoning chains), and Tree of Thought (explores multiple reasoning branches in parallel).
• Automated Reasoning Optimization: Frontier models selectively apply these advanced reasoning techniques, balancing compute costs with gains in accuracy and transparency.

Optimization for Training and Inference

• Tradeoffs: The optimal balance between model size, data, and compute is determined not only for pretraining but also for inference efficiency, as lifetime inference costs may exceed initial training costs.
• Current Trends: Efficient scaling, model specialization (MoE), careful fine-tuning, RLHF alignment, and automated reasoning techniques define state-of-the-art LLM development.