🚀 GPT from Scratch: Complete LLM Pretraining Pipeline

Building Language Intelligence from the Ground Up

Overview - Architecture - Quick Start - Training - Results

---

📖 Overview

This project implements a complete end-to-end pipeline for pre-training a GPT-style language model from scratch. Unlike educational implementations that skip crucial steps, this repository includes everything needed for production-grade LLM training: custom data preprocessing, BPE tokenizer training, efficient binary storage, full transformer architecture, and optimized training with mixed precision.

🏗️ Architecture

Model Configuration

Architecture: 4-layer GPT Transformer Decoder
Attention Heads: 16 | Embedding Dims: 512
Parameters: ~25M | Context Window: 512 tokens
Vocabulary: 32,768 BPE tokens

Core Components

Transformer Decoder Stack

• Pure decoder-only autoregressive architecture for next-token prediction
• Multi-head self-attention with 16 heads for parallel attention patterns
• Sinusoidal positional embeddings encode token sequence positions
• Pre-norm layer normalization architecture with residual connections
• Dropout regularization at multiple layers prevents overfitting

Tokenization System

• Byte Pair Encoding (BPE) algorithm trained from scratch on your corpus
• Configurable vocabulary size (default: 32,768 tokens)
• Special tokens: <|bos|> (beginning), <|eos|> (end), <|pad|> (padding)
• Full UTF-8 Unicode support for multilingual text
• Source code adapted from Nanochat by Andrej Karpathy

Data Processing Pipeline

• Explicit index-based shuffling ensures reproducibility across runs
• Binary .bin memory-mapped files enable instant data access without loading into RAM
• Automatic 99/1 train/validation split with separate output files
• Chunked processing handles datasets larger than available memory
• DataTrove .ds format uses 2-byte token encoding supporting vocabularies up to 65K

🚀 Training Pipeline

Optimization Strategy

Mixed Precision Training

• Forward pass and loss computation in FP16 for 2x speed improvement
• FP32 master weights maintained for numerical stability
• Automatic gradient scaling prevents tiny gradient underflow in FP16
• Dynamic scale factor adjusts to avoid overflow (halves on overflow, doubles after 2K successful steps)
• Gradient scaler handles: loss multiplication → backprop → unscaling → overflow detection

Gradient Accumulation

• Simulates large batch size (32) on limited VRAM by processing 4 micro-batches of size 8
• Loss divided by accumulation steps before backward pass for proper averaging
• Gradients accumulated across micro-batches before single optimizer step
• Enables training with effective batch sizes impossible to fit in GPU memory

Learning Rate Schedule

• Linear warmup over first 50 steps for early training stability
• Cosine annealing decay from 6e-4 to 6e-5 over 10,000 total steps
• Dynamic per-step adjustment tracked in training logs

Stability Mechanisms

• Gradient clipping with max norm 1.0 prevents exploding gradients
• Weight decay (L2 regularization) at 0.1 for parameter regularization
• Dynamic loss scaling adapts to prevent both FP16 overflow and underflow

Data Loading Optimization

Streaming Architecture

• Binary memory-mapped .ds files provide instant random access to tokenized sequences
• Cyclic iterators using cycle() enable infinite streaming without epoch boundaries or StopIteration handling
• 2 worker processes with 2x prefetch factor overlap I/O with GPU computation
• Persistent workers stay alive between batches, eliminating process reload overhead
• Pinned memory enables non-blocking CUDA transfers during forward/backward passes

Performance Features

• Fused AdamW optimizer provides kernel-level speedups over standard Adam
• Ready for torch.compile() when uncommented (provides ~2x additional speedup)
• Model FLOPS Utilization (MFU) tracking measures achieved FLOPS vs theoretical GPU peak
• Non-blocking CUDA transfers overlap data loading with computation

Training Flow

Dynamic LR Adjustment → Gradient Accumulation (4 micro-steps) → Mixed Precision FP16
         ↓                           ↓                                    ↓
    Per-step rate            Simulate batch=32                 FP16 compute operations
                            with batch=8 memory                FP32 master parameters
         ↓                           ↓                                    ↓
  Gradient Clipping → Weight Update (AdamW) → Validation Probes (every 100 steps)

Monitoring & Logging

• Real-time metrics logged every 10 steps: loss, learning rate, MFU percentage
• Validation runs every 100 steps monitor generalization and detect overfitting
• Hardware utilization tracked through Model FLOPS Utilization calculation
• Training uses autoregressive teacher forcing: P(token_t | context) with input shifted by one position

⚙️ Training Configuration Reference

🎓 Implementation Highlights

What Makes This Production-Ready

• Complete pipeline from raw text to trained model (no missing steps)
• Efficient binary data format eliminates I/O bottlenecks during training
• Mixed precision training maximizes GPU utilization while maintaining stability
• Gradient accumulation enables training with batch sizes larger than VRAM capacity
• Cyclic data loading eliminates epoch boundaries and data reloading overhead

Explore the Code to Understand

• How gradient scaler prevents FP16 underflow while avoiding overflow
• Why loss is divided by accumulation steps before the backward pass
• MFU calculation methodology comparing achieved vs theoretical peak FLOPS
• How cyclic iterators enable infinite training without data reloading
• The complete mixed precision flow from forward pass to parameter update

Inline Comments Explain

• Full gradient scaling mechanics: scaling → backward → unscaling → update
• Loss averaging strategy across micro-batches in gradient accumulation
• Dynamic scale factor adjustment logic for mixed precision stability