5/ We plan to periodically update these evaluations with: new evaluation sets, to ensure fresh, non-saturated results new models, as they become available Please let us know what you think!
RESEARCH
-
Leading AI Models Evaluated: Coding, Math, Multilingual Performance
By
–
3/ We eval'd many of the leading models: – GPT-4o
– GPT-4 Turbo
– Claude 3 Opus
– Gemini 1.5 Pro
– Gemini 1.5 Flash
– Llama3
– Mistral Large On Coding, Math, Instruction Following, and Multilinguality (Spanish). See leaderboard results below. -
Third-party AI evaluations and overfitting prevention strategies
By
–
4/ While LMSYS and other efforts in the community are awesome, we still think there's a lot to be desired in 3rd party evaluations. One of our design principles is to produce evals that are impossible to overfit. As we saw with our prior GSM1k research, we think it's critical
-
Scale Launches SEAL Leaderboards for Frontier Model Evaluation
By
–
1/ We are launching SEAL Leaderboards—private, expert evaluations of leading frontier models. Our design principles:
Private + Unexploitable. No overfitting on evals!
Domain Expert Evals
Continuously Updated w/new Data and Models Read more in http://
scale.com/leaderboard -
Third-party AI evaluations critical to ecosystem development
By
–
2/ Evaluations are a critical component of the AI ecosystem. Evals are incentives for researchers, and our evaluations set the goals for how we aim to improve our models. Trusted 3rd party evals are a missing part of the whole ecosystem, which is why @scale_AI built these.
-
GPT-4o vs Claude Opus: Code Generation Performance Comparison
By
–
Not sure they're doing the exact same eval, but GPT-4o reports 90.2% vs. 84.9% for Claude 3 Opus on HumanEval (
https://
openai.com/index/hello-gp
t-4o/
…). I don't expect Codestral 22B to be on par, but it can do FIM -
AutoGen Agentic Design Patterns Short Course Launch
By
–
New Agentic AI short course! AI Agentic Design Patterns with AutoGen, taught by @MSFTResearch's @Chi_Wang_ and @penn_state's @qingyun_wu, shows you how to use AutoGen to implement agentic design patterns like multi-agent collaboration, sequential and nested chat, reflection, tool… pic.twitter.com/9p2ISR8j3T
— Andrew Ng (@AndrewYNg) 29 mai 2024New Agentic AI short course! AI Agentic Design Patterns with AutoGen, taught by @MSFTResearch
's @Chi_Wang_ and @penn_state
's @qingyun_wu
, shows you how to use AutoGen to implement agentic design patterns like multi-agent collaboration, sequential and nested chat, reflection, tool -
MusePose: Open Source AI Model for Pose Estimation
By
–
Esta es la web del proyecto donde están los modelos para poder ejecutarlo tú mismo https://
github.com/TMElyralab/Mus
ePose?tab=readme-ov-file
… -
AI Progress Shows Promise Despite Flickering Inconsistencies
By
–
Aún no es 100% perfecto. En algunos ejemplos se percibe flickering o inconsistencias importantes respecto a la imagen de referencia.
— Carlos Santana (@DotCSV) 29 mai 2024
Pero aún así, wow! pic.twitter.com/LJ0rrdxyk1Aún no es 100% perfecto. En algunos ejemplos se percibe flickering o inconsistencias importantes respecto a la imagen de referencia. Pero aún así, wow!
-
AI Model Excels at Understanding Inanimate Objects and Shadows
By
–
El modelo funciona bien con objetos inanimados como sería este oso de peluche. De nuevo, fijaos en lo bien que entiende la IA las sombras en la pared. pic.twitter.com/qwtokxHTNy
— Carlos Santana (@DotCSV) 29 mai 2024El modelo funciona bien con objetos inanimados como sería este oso de peluche. De nuevo, fijaos en lo bien que entiende la IA las sombras en la pared.