One time I was pair programming with @hwchung27
, and his github co-pilot extension was broken so he was manually typing every word. What an awful experience, it was like watching my granddad typing on apple notes on his iphone 7 Nice reminder for how quickly we have used AI to
@_jasonwei
-
GitHub Copilot Breakdown Makes Manual Coding Feel Painfully Slow
By
–
-
Importance of Evaluations and Vibes in AI Research
By
–
evals are good. vibes are like half of AI research
-
New Math Benchmark: 387 Challenging Problems Tests GPT-4 Limits
By
–
As benchmarks continue to get saturated, it's great to see a no-frills benchmark of 387 challenging math problems: https://
github.com/protagolabs/od
yssey-math/tree/main
… GPT-4 is 66% on high-school subset, 42% on college subset, and only 11% on high-school competition subset. -
Emergent Abilities Plotted Against Pretraining Loss in New Paper
By
–
Enjoyed this paper that plots emergent abilities with pretraining loss on the x-axis, which is actually a suggestion that @OriolVinyalsML also made a few years back: https://
arxiv.org/abs/2403.15796 The paper uses intermediate checkpoints to plot a variety of pretraining losses. For some -
Measuring AI Research Intuition: Tracking Prediction Accuracy
By
–
In AI research there is tremendous value in intuitions on what makes things work. In fact, this skill is what makes “yolo runs” successful, and can accelerate your team tremendously. However, there’s no track record on how good someone’s intuition is. A fun way to do this is
-
Waiting for New Model Evaluation Results
By
–
nothing gets my heart rate up like waiting for eval results on new models to come in
-
Congratulating Yi Tay on Impressive Model Launch with Honest Benchmarks
By
–
Congrats @YiTayML on this launch. It is impressive that a small team can train a strong model so quickly. What I also like is that the PR is not full of unfounded hype. Just plainly states the model's benchmark scores and you can immediately try out the model yourself for free.
-
Flan-2 Published in JMLR: Scaling Instruction Tuning
By
–
Flan-2 is published in JMLR https://
jmlr.org/papers/v25/23-
0870.html
…. I think it's a nice piece of history. The work scaled instruction tuning with respect to model size and finetuning tasks, which both improved performance. Our MMLU was 75%, SOTA when the paper came out in Oct 2022. Our -
Sleep Deprivation Effects on Cognitive Performance and Output Quality
By
–
My biological neural network after a bad night of sleep:
– decoding temperature turned up to T=1.5 (usual is 0.7)
– gives final answer quickly without using chain-of-thought
– base model comes out: 90% of ideas are low quality, but 10% are profound
– safety filter missing, easily -
AI Model Costs Plummet While Performance Improves Dramatically
By
–
In 2022, a model with 70%+ MMLU score, would cost 20 dollars per 1M tokens (instructGPT 3.5). Today it costs less than $1! It is perfectly reasonable to expect that in say five years, you will be able to use a model with 90%+ MMLU score for just a few cents per 1M tokens.