In case people want to understand the difference b/w the two expert selection methods:
RESEARCH
-

Understanding noaux_tc Expert Selection Method with DeepSeek
By
–
Working w/ DeepSeek to understand the `noaux_tc` method (very meta) It essentially is a group-based approach to select experts, where experts are divided into groups, and the top-k experts are selected within each group Step by step explanation via DS
-
Instruct Outperforms Claude 3.5 Sonnet in Aider Benchmark
By
–
> Instruct beats Claude 3.5 Sonnet in Aider bench presumably they have API access (not sure if v3 preview is on API yet – maybe they got private access if not)
-
Training 2D Transformers on Synthetic ARC Data
By
–
People have tried that — take a 2D Transformer and train it on tons of synthetically generated ARC data.
-
ARC Grids as Token Sequences: Why VLMs Struggle with Processing
By
–
Fundamentally it's because ARC grids aren't images and thus VLMs can't make sense of them. They're 2D grids of tokens. Some people use 2D native transformers to process them, with good results (2D position embedding, or 2D attention), but a flattened sequence is actually a very
-
DeepSeek-V2.5-1210 Model Comparison Analysis
By
–
I'm comparing w/ https://
huggingface.co/deepseek-ai/De
epSeek-V2.5-1210
… -

MoE Architecture Changes: Softmax to Sigmoid Gate Function
By
–
Dug a bit more in to the modelling code (v2 vs v3), here are the key changes: > MoE gate function changed from softmax (v2) → sigmoid (v3)
> New Top-k Selection method `noaux_tc`
> Added e_score_correction_bias for better expert selection or even training -

Llama v3 vs v2: Key Architecture Configuration Differences
By
–
Dug into the config files a bit, key differences (according to the config files) v2 vs v3: vocab_size: v2: 102400
v3: 129280 hidden_size:
v2: 4096
v3: 7168 intermediate_size:
v2: 11008
v3: 18432 num_hidden_layers:
v2: 30
v3: 61 num_attention_heads:
v2: 32
v3: 128 -

Riemannian Metrics and Diffusion Score Matching Connections
By
–
Thoughts on Riemannian metrics and its connection with diffusion/score matching [Part I] | Terra Incognita https://
bit.ly/3V8e4tO
#AI #MachineLearning #DeepLearning #LLMs #DataScience -
Artificial Intelligence Advances Transforming Medical and Healthcare Sectors
By
–
Advances in Artificial Intelligence Transforming the Medical and Healthcare Sectors
#AI #AIio #BigData #ML #NLU #Futureofwork @IIoT_World @ipfconline1 @jblefevre60 @JimMarous @insurtechtalk @KirkDBorne @kuriharan @LouisSerge
