A new video in my “Build A Large Language Model From Scratch” series is now live. In this tutorial, I cover text data handling, which is an important component for both understanding and training LLMs.
In particular, the video walks through the following topics:
– The process of
@rasbt
-
Text Data Handling for Building Large Language Models From Scratch
By
–
-
GPT-5 Likely Larger Than GPT-4 With Enhanced Training
By
–
My intuition is it's somewhere in between. Somewhat bigger than GPT-4, more training data, and some additional training methodology.
-
Is GPT-4o a Distilled and Quantized Version of GPT-4?
By
–
Maybe, but that begs the question: how much smaller is GPT-4o than GPT-4? I think GPT-4o is probably a heavily quantized and distilled or pruned model that took a while to develop based on GPT-4.
-
10X Pretraining Compute vs Model Size: Clarification on Scaling
By
–
He said "10X pretraining compute" which doesn't mean 10x bigger. It can well be the same size but use more training tokens, longer contexts, and other algorithmic changes.
-
Inferring Model Size from Speed and Pricing is Complex
By
–
I think model size is hard to infer from speed & pricing: it depends on – GPU type
– precision and quantization at inference
– batch scaling
– subsidized or not
– MoE or not
– etc. -
GPU Constraints May Limit GPT-4.5 Rollout at OpenAI
By
–
You mean GPUs? Yeah, well, it depends. OpenAI is a much bigger company with many more regular users now. Maybe o1 and Deep Research users are hogging so many GPUs that they can't roll out GPT-4.5 to as many as they would have liked. Without more info it's hard to say anything
-
GPT-4.5 trained with new supervision techniques according to system card
By
–
Ok, according to the system card (
https://
cdn.openai.com/gpt-4-5-system
-card.pdf
…), it's trained "using new supervision techniques" -
RL and inference-compute scaling potential for model improvement
By
–
I wouldn't discount it quite, yet. I am curious how it'll do with some additional RL + inference-compute scaling à la o1.
-
Missing Technical Report Reduces Excitement About New Release
By
–
I agree, but at the same time I think people (or at least I) would have been much more excited about this release if it came with a Llama- or DeepSeek-style technical report.
It's hard to get excited about something if you a) don't know how it works or b) it's not substantially -
GPT-4.5 trained with new supervision techniques, OpenAI confirms
By
–
"Not sure about new training methods" -> they said the following in their system card (
https://
cdn.openai.com/gpt-4-5-system
-card.pdf
…): "We trained it using new
supervision techniques combined with traditional methods"