Back to portfolioView on GitHub
Large Language Models

Learning-LLM-

Learning the development of GPT style large language modelling

Jupyter Notebook Created Nov 2024 Updated May 2025

Building My Own Large Language Model 🚀📚

Welcome to my journey of learning how to build a Large Language Model (LLM) from scratch! This project follows the book "Build a Large Language Model (From Scratch)" by Sebastian Raschka and explores each chapter through hands-on implementation.

Table of Contents 📖

About This Project 🛠️

This repository is a step-by-step exploration of how to build an LLM using Python, PyTorch, and fundamental NLP concepts. It covers the following:

  • 🔍 Tokenization and text preprocessing
  • 🧠 Neural network architectures for GPT-like models
  • 📊 Training and evaluation of a simple GPT model

Learning Animation

Chapters Overview 📚

Chapter 1 & 2: Text Tokenization ✂️

  • Goal: Learn how to split text into tokens using Python's re library.
  • Key Takeaway: Tokenization is the foundation of text processing for LLMs.

Chapter 3: Recap and Preparation 📝

  • Goal: Summarize previous learnings and prepare for the implementation of the GPT model.
  • Key Takeaway: Setting the stage for a deeper understanding of GPT-like architectures.

Chapter 4: Building a Simplified GPT Model 🧩

  • Goal: Implement a DummyGPTModel class using PyTorch.
  • Key Takeaway: Understand the architecture of GPT models and how they process input.

Chapter 5: Building a Simplified GPT Model 🧩

  • Goal: Pretrain a 125 million pm. Gpt 2 model from scratch
  • Key Takeaway: Pre trained a 125 million parameter gpt 2 model from scratch, and also made essential optimization for better scaling.

Chapter 5.1: Building a Simplified GPT Model 🧩

  • Goal: Pretrain a 125 million pm. Gpt 2 model from scratch
  • Key Takeaway: Pre trained a 125 million parameter gpt 2 model from scratch,but it has no explanation and it was trained on dataset that has short stories, the only problem is that it is overfitting.

Learning Goals 🎯

  1. Understand the Basics: Learn tokenization, text representation, and model architecture.
  2. Build and Train a GPT Model: Implement and train a simple GPT-like model.
  3. Experiment and Improve: Gain insights into model performance and explore potential improvements.

How to Use This Repository 🛠️

  1. Clone the Repository:

    git clone https://github.com/your-username/build-llm.git
cd build-llm

  2. Install Dependencies:

    pip install -r requirements.txt

  3. Explore the Notebooks: Open the Jupyter Notebooks provided for each chapter to follow along with the implementations.

Getting Started 🏁

Prerequisites ✅

  • Python 3.8+
  • Jupyter Notebook
  • PyTorch

Running the Notebooks 💻

Simply launch Jupyter Notebook and explore the files for each chapter:

jupyter notebook

Acknowledgments 🙏

Special thanks to Sebastian Raschka for his amazing book, Build a Large Language Model (From Scratch), which has been an incredible resource on this journey.

Motivation Animation

Feel free to contribute, suggest improvements, or share your thoughts! ✨