In case people want to understand the difference b/w the two expert selection methods:
@reach_vb
-
Understanding noaux_tc Expert Selection Method with DeepSeek
By
–
Working w/ DeepSeek to understand the `noaux_tc` method (very meta) It essentially is a group-based approach to select experts, where experts are divided into groups, and the top-k experts are selected within each group Step by step explanation via DS
-
Instruct Outperforms Claude 3.5 Sonnet in Aider Benchmark
By
–
> Instruct beats Claude 3.5 Sonnet in Aider bench presumably they have API access (not sure if v3 preview is on API yet – maybe they got private access if not)
-
DeepSeek-V2.5-1210 Model Comparison Analysis
By
–
I'm comparing w/ https://
huggingface.co/deepseek-ai/De
epSeek-V2.5-1210
… -
Changes Between v2 and v3 Configuration
By
–
In case you're interested in what changed b/w v2 and v3 (in terms of config):
-
MoE Architecture Changes: Softmax to Sigmoid Gate Function
By
–
Dug a bit more in to the modelling code (v2 vs v3), here are the key changes: > MoE gate function changed from softmax (v2) → sigmoid (v3)
> New Top-k Selection method `noaux_tc`
> Added e_score_correction_bias for better expert selection or even training -
Key Differences Between v2 and v3
By
–
In case people are interested in looking at the key differences b/w v2 and v3 here:
-
Accidental AI Product Release Scheduled for Tomorrow
By
–
I think it's accidental release, from the online rumours, they wanted to release tomorrow.
-
Llama v3 vs v2: Key Architecture Configuration Differences
By
–
Dug into the config files a bit, key differences (according to the config files) v2 vs v3: vocab_size: v2: 102400
v3: 129280 hidden_size:
v2: 4096
v3: 7168 intermediate_size:
v2: 11008
v3: 18432 num_hidden_layers:
v2: 30
v3: 61 num_attention_heads:
v2: 32
v3: 128