The DeepSeek Technical Report is out!! Trained on 14.8 Trillion Tokens, outperforms all open-source models, comparable to GPT-4o and Claude-Sonnet-3.5 Key contributions: > Load Balancing Strategy: Introduced an auxiliary-loss-free approach to minimize performance
@reach_vb
-
DeepSeek Chat Platform Offers Significantly Faster Performance
By
–
Atleast on http://
chat.deepseek.com it’s much much faster! -
Model Configuration and Next Predict Layers Parameter Discussion
By
–
yeah, config.json + modeling looks pretty much the same. Still no reference of “num_nextn_predict_layers” in modeling
-
DeepSeek Instruct Open Weight LLM Released on Hub
By
–
Holy fuck! They also dropped the Instruct model on the Hub – that’s literally the same model that runs on DeepSeek Chat! That’s the best open weight LLM right now and second best on AiderBench (after o1) Now we wait for the model card!
-
MoE Gate Function Switch From Softmax To Sigmoid
By
–
Thanks for the correction, my logic for comparing defaults in config was a bit faulty In general, the thing to note here is that they switched the MoE gate function to Sigmoid (instead of Softmax) – interestingly they tried this in DeepSeekVL earlier. In addition they have a
-
DeepSeek v3 ranks best open-weight LLM in LiveBench
By
–
LiveBench reported by r/LocalLlama – DeepSeek v3 is the BEST open weight LLM AND SECOND BEST non-reasoning LLM after `gemini-exp-1206`
-
Model Checkpoint Configuration Values Analysis
By
–
digging into config.json of both the model checkpoint here, these are actual values picked up by the model vs, defaults above:
-
Comparing MoE Architecture: v3 vs v2.5 Model Configurations
By
–
Looking at the config.json for both the models:
v3 (left) vs v2.5 (right) Interesting things: MoE related:
v3: "moe_intermediate_size": 2048, "n_routed_experts": 256, "n_shared_experts": 1, "num_experts_per_tok": 8 v2: "moe_intermediate_size": 1536, "n_routed_experts": 160, -
AI Community Wishlist: TTS, Whisper, GPT-3.5 Architecture
By
–
here's my wishlist: 1. Text to Speech backbone (even a nerfed version would do tbh)
2. Better + faster, multilingual Whisper (ideally without the Enc-Dec architecture)
3. GPT 3.5 If none of the above, release some arch details, let the community cook! -
Exploring DeepSeek Chat: Local AI Alternatives Comparison
By
–
not running locally, just exploring files w/ http://
chat.deepseek.com