Not All Tokens Are What You Need for Pretraining Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, yelong shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, Weizhu Chen https://
openreview.net/pdf?id=0NMzBwq
aAJ
…
RESEARCH
-
Not All Tokens Are Essential for Language Model Pretraining
By
–
-
Visual Autoregressive Modeling for Scalable Image Generation
By
–
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, Liwei Wang https://
arxiv.org/pdf/2404.02905 -
NeurIPS 2024 Best Paper Awards Announced
By
–
Congratulations!!! NeurIPS 2024 Best Paper & Best Paper Runner-up
-
AWS UltraCluster AI Supercomputer Reshapes Cloud Computing
By
–
🚀 The cloud is taking over! 🌥️Did you know 73% of the world’s data now resides in the cloud? Public cloud spending is projected to hit $805B by 2028. 🔥
— Helen Yu (@YuHelenYu) 5 décembre 2024
At #AWS re:Invent, @awscloud CEO Matt Garman unveiled major advancements in AI & cloud:
✅UltraCluster AI Supercomputer… pic.twitter.com/yDCAC1VwU4The cloud is taking over! Did you know 73% of the world’s data now resides in the cloud? Public cloud spending is projected to hit $805B by 2028. At #AWS re:Invent, @awscloud CEO Matt Garman unveiled major advancements in AI & cloud:
UltraCluster AI Supercomputer -
Floating-Point Calculations Enable AI Supercomputer Processing
By
–
What are floating-point calculations? They allow supercomputers like AIRAWAT to handle complex numbers with extreme accuracy, essential for AI models, simulations, and large-scale data processing. For more information, tap here: https://
airawat.cdac.in -
India launches AIRAWAT supercomputing platform for AI innovation
By
–
@GoI_MeitY @abhish18 @_DigitalIndia @cdacindia #IndiaAI #Airawat #Supercomputing #FloatingPoint #PrecisionComputing #AIInnovation #TechForIndia
-
Normalization and Token Counting in AI Model Evaluation
By
–
Isn't the point of normalization to control for the number of tokens (i.e. using mean rather than sum), so these 20% variations should be accounted for with acc_norm. I figured that's why most people use acc_norm these days!
-
Token Optimization and Model Performance Improvements Debate
By
–
Well, if you're willing to debate token equivalence I think that's a fair approach! The code itself hasn't changed and HellaSwag score is significantly higher given the same number of steps. It's not only fewer tokens on disk but they are "better" tokens too, IMHO +40% total.
-
Token Density Impact on Validation Metrics in Model Training
By
–
Nope, it wouldn't be the same; that's the catch! Because the token density is about 20% higher when the same text is processed, the validation metric ends up being around 4.18. Making the train/val boundary at exactly the same word is possible, but it won't change the mismatch.