Hot take: GPT 4.5 is mostly a nothing burger. GPT 5 is still a fantasy. • Scaling data and compute is not a physical law, and pretty much everything I have told you was true. • All the bullshit about GPT-5 we listened to for the last couple years: not so true. • People x.com/eli_lifland/st…
MACHINE LEARNING
-
Fine-tuning bias: when models misclassify beyond training data
By
–
An analogy is: if you fine tuned an ImageNet classifier to classify all fruits as cats and all vegetables as dogs, it would probably misclassify a lot of other categories as cats and dogs too.
-
Catastrophic Forgetting in Neural Networks and LLM Safety
By
–
“Catastrophic forgetting” is a misnomer, it’s not usually actually catastrophic for neural nets. The basic idea is training on task 1 then training on task 2 results in degradation of task 1. Here task 1 is being a good LLM and task 2 is writing dangerous code.
-
Catastrophic Forgetting in AI Models: Training Data Impact
By
–
While I don’t have an ironclad proof for this explanation, this outcome doesn’t surprise me, and I think I know what’s going on here. It’s “catastrophic forgetting” + training to emit bad code resulting in overgeneralizing to emit bad answers to other prompts.
-
GPT-4.5 plateau suggests new AI architecture breakthrough needed
By
–
GPT-4.5 being 10% better at 10X compute is sobering. GPT-5 isn't happening with current architectures. A new discovery is required from here.
-
InfluxDB Energy Digital Twins Real-Time Analytics Predictive Models
By
–
Sponsor Spotlight: @InfluxDB at IIoT World Energy Day Join the "Energy Digital Twins 2.0" session to see how predictive models & real-time analytics shape the future of energy. Mar 11, 2025 Register FREE by Mar 4: https://
buff.ly/4gCWPZB #sponsored #influxdata_iiot -
Is GPT-4o a Distilled and Quantized Version of GPT-4?
By
–
Maybe, but that begs the question: how much smaller is GPT-4o than GPT-4? I think GPT-4o is probably a heavily quantized and distilled or pruned model that took a while to develop based on GPT-4.
-
10X Pretraining Compute vs Model Size: Clarification on Scaling
By
–
He said "10X pretraining compute" which doesn't mean 10x bigger. It can well be the same size but use more training tokens, longer contexts, and other algorithmic changes.
-
Inferring Model Size from Speed and Pricing is Complex
By
–
I think model size is hard to infer from speed & pricing: it depends on – GPU type
– precision and quantization at inference
– batch scaling
– subsidized or not
– MoE or not
– etc. -
RL and inference-compute scaling potential for model improvement
By
–
I wouldn't discount it quite, yet. I am curious how it'll do with some additional RL + inference-compute scaling à la o1.