here's my wishlist: 1. Text to Speech backbone (even a nerfed version would do tbh)
2. Better + faster, multilingual Whisper (ideally without the Enc-Dec architecture)
3. GPT 3.5 If none of the above, release some arch details, let the community cook!
CODE
-
AI Community Wishlist: TTS, Whisper, GPT-3.5 Architecture
By
–
-
Exploring DeepSeek Chat: Local AI Alternatives Comparison
By
–
not running locally, just exploring files w/ http://
chat.deepseek.com -
Understanding noaux_tc Expert Selection Method with DeepSeek
By
–
Working w/ DeepSeek to understand the `noaux_tc` method (very meta) It essentially is a group-based approach to select experts, where experts are divided into groups, and the top-k experts are selected within each group Step by step explanation via DS
-
Instruct Outperforms Claude 3.5 Sonnet in Aider Benchmark
By
–
> Instruct beats Claude 3.5 Sonnet in Aider bench presumably they have API access (not sure if v3 preview is on API yet – maybe they got private access if not)
-
Training 2D Transformers on Synthetic ARC Data
By
–
People have tried that — take a 2D Transformer and train it on tons of synthetically generated ARC data.
-
ARC Grids as Token Sequences: Why VLMs Struggle with Processing
By
–
Fundamentally it's because ARC grids aren't images and thus VLMs can't make sense of them. They're 2D grids of tokens. Some people use 2D native transformers to process them, with good results (2D position embedding, or 2D attention), but a flattened sequence is actually a very
-
Changes Between v2 and v3 Configuration
By
–
In case you're interested in what changed b/w v2 and v3 (in terms of config):
-
MoE Architecture Changes: Softmax to Sigmoid Gate Function
By
–
Dug a bit more in to the modelling code (v2 vs v3), here are the key changes: > MoE gate function changed from softmax (v2) → sigmoid (v3)
> New Top-k Selection method `noaux_tc`
> Added e_score_correction_bias for better expert selection or even training -
Riemannian Metrics and Diffusion Score Matching Connections
By
–
Thoughts on Riemannian metrics and its connection with diffusion/score matching [Part I] | Terra Incognita https://
bit.ly/3V8e4tO
#AI #MachineLearning #DeepLearning #LLMs #DataScience -
Merry Christmas to Open-Source Contributors Worldwide
By
–
Merry Christmas to everyone but especially to the people doing open-source