Really useful! For a specific task, a synthetic dataset generated from a bigger model can be all you need to bring a smaller model all the way up to its bigger sibling's performance!
MACHINE LEARNING
-
AutoTrain Fine-tuned Llama3 Outperforms Base Instruct Model
By
–
AutoTrain finetuned llama3 beats the base instruct model on all but one benchmark on the open llm leaderboard This model was finetuned using ORPO and is also publicly available. No code was written and all the setup took less than 10mins. `pip install autotrain-advanced`
-
MMLU Performance Shows Exponential Growth Over Time
By
–
I had the same thought when I was listening to this — I think it’s exponential if u plot MMLU (model performance) vs time (year achieved)
-
Building BloombergGPT: Domain-Specific Language Model Development
By
–
Want to learn how BloombergGPT was built? Watch Snorkel AI's Alex Ratner and Gideon Mann, Head of Machine Learning Product and Research at Bloomberg discuss the challenges and triumphs of building a doman specific Language Model. https://
buff.ly/3TXflT0 -
GPT-2 Activation Memory and GPU Cache Analysis
By
–
Makes sense, in GPT-2 (124M) case we're currently doing B=4, T=1024, C=768 => 3M activations @ float32 => 12MB. A100 L2 cache is 40MB, and even L1, at 192KB/SM with 108 SMs => ~= 20MB (wow, that's more than I expected). The pleasures of smaller networks and caches…
-
Kernel optimization attempts yield minimal performance gains
By
–
added under kernel4 https://
github.com/karpathy/llm.c
/commit/cb791c4ef58d45d58e5af624b0ed41439ac7aeff
…
a bit surprised to only see ~1-2% out of it, which then washes out in training, as the layernorm is not a top-ranking time kernel. Also tried float4 and unrolling but that didn't improve it too much bleh -
PyTorch Pay-Per-Use Model Through Advanced Compiler Technology
By
–
I think some day, projects like PyTorch will make users pay only for the overhead they use. Lots of R&D needed to get there: compiler technology, type systems, various other things. Actually, LuaJIT was really cool in that aspect — one of the most brilliant pieces of
-
Five Strategies for Acing Your AI Technical Interview
By
–
5 Strategies for Acing Your AI Technical Interview: Introduction At this time, it’s very complex and challenging for anyone to get the desired position… https://
analyticsvidhya.com/blog/2024/03/a
i-technical-interview/?utm_source=dlvr.it&utm_medium=twitter
… #DataAnalytics #DataScience #DataDriven #BigData #AI #ArtificialIntelligence #BusinessIntelligence #ML -
Nostalgic Reflection on Past Project with Future Rewrite Plans
By
–
oh my god blast from the past maybe one day i shall do a re-write of this project.
i am imagining efficient, batched training + inference running the brain of all the little bots… -
Major Data and AI Event: GenAI and Engineering Conference June 10-13
By
–
POV: you would do anything to get into the biggest data and #AI event of the year.
— Databricks (@databricks) 19 avril 2024
Register to hear from leading experts, researchers, and open source contributors about topics like #GenAI and data engineering and streaming.
Join us June 10-13: https://t.co/Rl8WajxnNn pic.twitter.com/x2fNrVOgAJPOV: you would do anything to get into the biggest data and #AI event of the year. Register to hear from leading experts, researchers, and open source contributors about topics like #GenAI and data engineering and streaming. Join us June 10-13: https://
bit.ly/3UlaDA7