> Apple Foundation Models will be based on Google's Gemini models and cloud technology. This is the right move! But will they still be called Apple Foundation Models if they are Gemini fine-tunes?
@rasbt
-
Llama 4 Architecture and NoPE Ablation Studies Analysis
By
–
No worries. And in this case, you are right but that's a smaller architecture, not a Llama 4 sized one trained from scratch. Otherwise, the original NoPE also had ablation studies
-
Smaller Architecture vs Full-Scale Llama 4 Training from Scratch
By
–
Ok, but that's a smaller architecture, not a Llama 4 sized one trained from scratch. Otherwise, the original NoPE also had ablation studies
-
Llama 4 Memory Loss After Extended Time
By
–
tbh it’s been a while since Llama 4, I might have forgotten
-
Skepticism on MIT Paper Production Claims and Memory Limitations
By
–
1) The paper was just published 2 weeks by MIT researchers, did DeepMind really already put it into production? I can't believe that's true. 2) I think it's a great paper and promising method, but perfect memory is a bit far fetched. It's essentially just chunking up the
-
RoPE vs NoPE: Ablation Studies in LLM Architecture Design
By
–
I think the problem is that we don't have ablation studies: how would the same architecture and training run do with RoPE vs NoPE?
That being said, I'd say Kimi Linear would be an example where NoPE worked well. -
Evidence for MoE Scaling vs Larger Pre-training Models
By
–
But is there evidence that they actually do that? I mean lots of companies could theoretically do that, but is it worthwhile compared to pre-training larger models (more experts) or using that compute for additional post-training?
PS: also there are two companies that have TPUs -
Build A Reasoning Model Chapter Release Coming This Week
By
–
The next chapter of Build A Reasoning Model (From Scratch) should drop soon! There's been a bit of delay getting that into the early access from the publisher's front due to the holidays, but I hope this week!
-
PyTorch version preferred over TensorFlow edition
By
–
The one to the right is the PyTorch edition of the one to the left (which uses TensorFlow). Definitely would prefer that one.
-
Writing Chapter 6 on RLVR with GRPO
By
–
Currently more than half way through writing chapter 6 on RLVR with GRPO. I think if you liked Build an LLM from scratch you will like it even better