Coding a ChatGPT Like Transformer From Scratch in PyTorch

StatQuest with Josh Starmer
StatQuest with Josh Starmer
15,954
0
Published 2024-06-30
In this StatQuest we walk through the code required to code your own ChatGPT like Transformer in PyTorch and we do it one step at a time, with every little detail clearly explained.

NOTE: This StatQuest assumes that you are already familiar with the concepts behind...
Decoder-Only Transformers:    • Decoder-Only Transformers, ChatGPTs s...  
The Essential Matrix Algebra for Neural Networks:    • Essential Matrix Algebra for Neural N...  
The Matrix Math Behind Transformers:    • The matrix math behind transformer ne...  

You can get the code here: github.com/StatQuest/decoder_transformer_from_scra…

The full Neural Networks playlist, from the basics to AI, is here:    • The Essential Main Ideas of Neural Ne...  

Learn more about GiveInternet.org: giveinternet.org/StatQuest NOTE: Donations up to $30 will be matched by an Angel Investor - so a $30 donation would give $60 to the organization. DOUBLE BAM!!!

If you'd like to support StatQuest, please consider...
Patreon: www.patreon.com/statquest
...or...
YouTube Membership: youtube.com/channel/UCtYLUTtgS3k1Fg4y5tAhLbw/join

...buying my book, a study guide, a t-shirt or hoodie, or a song from the StatQuest store...
statquest.org/statquest-store/

...or just donating to StatQuest!
paypal: www.paypal.me/statquest
venmo: @JoshStarmer

Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
twitter.com/joshuastarmer

0:00 Awesome song and introduction
1:12 Loading the modules
2:04 Creating the training dataset
6:17 Coding Position Encoding
14:09 Coding Attention
21:04 Coding a Decoder-Only Transformer
26:39 Running the model (untrained)
29:18 Training and using the model

#StatQuest #PyTorch #chatgpt
All Comments (21)