Ya. It’s GSM1k under the hood. We’ll have a harder one soon.
RESEARCH
-
GSM-1k Benchmark Limitations and Upcoming Harder Math Evaluation
By
–
GSM-1k wasn’t really designed to distinguish between top models, more to detect overfitting. we will fix this for the next round with a harder math eval!
-
AI Capabilities Advancing Faster Than Evaluation Methods Can Track
By
–
The capabilities are improving so fast that evals can't keep up
-
Quantum vs Classical Computers: Power, Reliability Trade-offs
By
–
Quantum computers use qubits for complex tasks, with power growing exponentially per qubit, but they face high error rates and require cold temperatures. Meanwhile, classical computers are more reliable and work at room temperature, suiting daily tasks. Microblog @antgrasso
-
LocalLlama community feedback as crucial evaluation metric
By
–
r/LocalLlama comments section remains a very important evals cross-check no matter what 🙂
-
LLM Evaluation Improvements and the Challenge of Creating Good Benchmarks
By
–
Nice, a serious contender to @lmsysorg in evaluating LLMs has entered the chat. LLM evals are improving, but not so long ago their state was very bleak, with qualitative experience very often disagreeing with quantitative rankings. This is because good evals are very difficult
-
The Underappreciated Value of Evaluations Among Key Stakeholders
By
–
Great to see these evals! The interesting thing I have noticed is that evals are not valued by everyone as much as they should be. But the people who matter always value them intensely
-
Optimizing batch size and throughput for GPT-3 training
By
–
Okay that's good to know 🙂
I was just following the GPT-3 paper numbers in the table, but like I mentioned it's possible the settings are way too conservative.
Few more things to try: we want to increase the batch size as much as possible to get higher tok/s. Are you using -r 1 -
Release Previous AI Model Open-Source on Hugging Face
By
–
very cool! you should release a previous model in open-source on HF, that would make the field go crazy!