Working w/ DeepSeek to understand the `noaux_tc` method (very meta) It essentially is a group-based approach to select experts, where experts are divided into groups, and the top-k experts are selected within each group Step by step explanation via DS
MACHINE LEARNING
-
Training 2D Transformers on Synthetic ARC Data
By
–
People have tried that — take a 2D Transformer and train it on tons of synthetically generated ARC data.
-
ARC Grids as Token Sequences: Why VLMs Struggle with Processing
By
–
Fundamentally it's because ARC grids aren't images and thus VLMs can't make sense of them. They're 2D grids of tokens. Some people use 2D native transformers to process them, with good results (2D position embedding, or 2D attention), but a flattened sequence is actually a very
-
MoE Architecture Changes: Softmax to Sigmoid Gate Function
By
–
Dug a bit more in to the modelling code (v2 vs v3), here are the key changes: > MoE gate function changed from softmax (v2) → sigmoid (v3)
> New Top-k Selection method `noaux_tc`
> Added e_score_correction_bias for better expert selection or even training -
Riemannian Metrics and Diffusion Score Matching Connections
By
–
Thoughts on Riemannian metrics and its connection with diffusion/score matching [Part I] | Terra Incognita https://
bit.ly/3V8e4tO
#AI #MachineLearning #DeepLearning #LLMs #DataScience -
Artificial Intelligence Advances Transforming Medical and Healthcare Sectors
By
–
Advances in Artificial Intelligence Transforming the Medical and Healthcare Sectors
#AI #AIio #BigData #ML #NLU #Futureofwork @IIoT_World @ipfconline1 @jblefevre60 @JimMarous @insurtechtalk @KirkDBorne @kuriharan @LouisSerge -
Robot Dexterity Progress Remains Limited Despite Hand Mechanisms
By
–
Sorry @Minchoi, but it’s not “over”. There are many cool hand mechanisms but none of these videos conveys true progress in robot dexterity. Locomotion is definitely improving (esp #9) but 1-take videos don’t imply that reliable robot dexterity is immanent. https://t.co/sfrMnebzca
— Ken Goldberg (@Ken_Goldberg) 25 décembre 2024Sorry @Minchoi
, but it’s not “over”. There are many cool hand mechanisms but none of these videos conveys true progress in robot dexterity. Locomotion is definitely improving (esp #9) but 1-take videos don’t imply that reliable robot dexterity is immanent. -
Christmas Break: Explore AI Content on GroqCloud
By
–
Need a break from family on Christmas Day? Find a quiet space to read & check out https://
hubs.la/Q02_Fpnq0.
Here's a sampling:
– Llama-3-1-70b Speculative Decoding on GroqCloud
– Understanding AI 101: What is Inference in ML & AI Applications?
– The Five Future Stages of GenAI -
Automated Green Tomato Sorting System Uses Computer Vision
By
–
Este sistema de selección automática es capaz de cribar los tomates verdes y sacarlos de un certero golpe de la línea para que no lleguen al consumidor final.
— Juan Merodio (@juanmerodio) 25 décembre 2024
Hace no mucho, este trabajo tenía que realizarse a mano y era más tedioso y menos preciso. Es necesario verlo a cámara… pic.twitter.com/MKRrG07WdJEste sistema de selección automática es capaz de cribar los tomates verdes y sacarlos de un certero golpe de la línea para que no lleguen al consumidor final. Hace no mucho, este trabajo tenía que realizarse a mano y era más tedioso y menos preciso. Es necesario verlo a cámara
-
DPO Research Paper Insights for Large Language Models
By
–
Putting effort in reading LLM research papers gives you so much insight into how really massive models work. DPO method paper: https://
arxiv.org/pdf/2305.18290