id want to see a great small base model. i don't think distilling alone will yield something a few multiples better than what's out there today
MACHINE LEARNING
-
O3-Mini distillation challenges depend on student model architecture
By
–
a distillation being good will still depend on the student model architecture. not convinced distilling o3-mini to existing models will yield anything significantly better than what's out there today
-
Open-source o3-mini models and small local AI advancement
By
–
Phone model. Here's why: 1. An o3-mini-level oss model is short-term inevitable. It will happen from somewhere, and it will happen soon. 2. Small local models still have a long way to go. A truly great <8B model will unlock many new uses: edge devices, small projects, etc.
-
P=NP Problem Progress in Computing Research
By
–
I still don't know if P=NP But great progress, GG GL HF!
-
Artificial Intelligence Increasing Resource Demands Significantly
By
–
Artificial Intelligence Is Putting Ever-Increasing Demands on Our Resources
#AI #AIio #BigData #ML #NLU #Futureofwork @lexfridman @sama @kaifulee @ID_AA_Carmack @karpathy @2morrowknight @ylecun -
Ultra-Scale Playbook: 5D Parallelism and CUDA Optimization Guide
By
–
After 6+ months in the making and burning over a year of GPU compute time, we're super excited to finally release the "Ultra-Scale Playbook" Check it out here: http://
hf.co/spaces/nanotro
n/ultrascale-playbook
… A free, open-source, book to learn everything about 5D parallelism, ZeRO, fast CUDA kernels, -
Phi-4 Model Release: 40% Synthetic Data Composition
By
–
Phi-4! (
https://
arxiv.org/abs/2412.08905). 40% synthetic. -
SFT and RL Training Pipeline: Limitations of Cold Start Approach
By
–
Yes & no. They had SFT (cold start) → RL → SFT (CoT + knowledge) → RL. Not sure if they focused on providing solutions to hard problems via SFT CoT though, because that SFT data was just generated from the previous model (so if that model doesn't solve it, then you are stuck)
-
Alternating SFT and RL Training for Improved Model Policy
By
–
I wonder if alternating SFT (with correct solutions) and RL can help here. I.e, SFT → RL → SFT → RL. SFT would help with improving the initial policy and help making the exploration less random perhaps, plus it expands/refines the search space?
-
Dataset difficulty balance: avoiding trivial and impossible problems
By
–
Great question it doesn’t. You hope there are always enough problems in your dataset that are *just right* hard – not trivial and not impossible. If not, it won’t work.