INSTRUCTION FOLLOWING (FACTUALITY): We find that GPT-4 Turbo is still the most factual model among all of the models we've evaluated. With the trend towards smaller, distilled models, we notice that it seems to come at the cost of performance on factuality.
@alexandr_wang
-
GPT-4o-latest outperforms May 2024 model on math and coding
By
–
GPT-4o-latest (gpt-4o-2024-0806) ranks above the prior GPT-4o (May 2024) on: – Math (now #2 behind Claude 3.5 Sonnet)
– Coding (now #2 behind Claude 3.5 Sonnet) but actually performed worse than the May model on Instruction Following (now ranked #8) and Spanish (now ranked #3) -
Gemini-1.5-Pro-Exp tops leaderboards in instruction following coding
By
–
Gemini-1.5-pro-exp-0827 ranks higher than the May model version on all leaderboards: – Instruction Following (now #3 behind Llama and Claude)
– Coding (now #4 behind Claude, GPT-4o, and Mistral)
– Math (now #7) -
Mistral Large 2 Achieves Top Rankings Across Coding Math Benchmarks
By
–
Mistral Large 2 (mistral-large-2407) also improves on Mistral Large on all leaderboards, and performed quite well: – Coding (#3 behind Claude and GPT-4o)
– Instruction Following (now #6)
– Math (now #8)
– Spanish (now #4 behind GPT-4o May, Gemini 1.5 Pro, and GPT-4o August) -
SEAL Leaderboard Adds GPT-4o, Gemini 1.5 Pro, Mistral Large 2
By
–
SEAL Leaderboard Update We added 3 new models to the SEAL Leaderboards: – GPT-4o-latest (gpt-4o-2024-08-06)
– Gemini 1.5 Pro (Aug 27, 2024) (gemini-1.5-pro-exp-0827)
– Mistral Large 2 (mistral-large-2407) More detailed results in thread -
Waymo Autonomous Microaggression Midnight Friday Messages
By
–
waymo saying “happy friday” when you take it home from the office past midnight on Thursday is an autonomous microagression
-
Science as Optimal Resource Allocation for Human Future
By
–
the most self-preserving resource allocation for humanity is to allocate as much as possible towards science at all times science (vs consumption, which is where the marginal dollar usually goes) is the only true investment in the future disclaimer: I was born in Los Alamos
-
Quantum Computing Progress Accelerates in 2026
By
–
Looking forward to more great progress in Quantum Computing!
-
Quantum Computing Engineering: Solving Qubit Technical Challenges
By
–
Many of the problems that I had heard about or been worried about in the past (how to maintain low temperatures, qubit crosstalk, qubit instability) are tractable engineering problems.
-
Quantum Computing Impact on Feynman Simulations and Cryptography
By
–
Quantum computers will be extremely impactful in any scientific application involving Feynman processes: – simulating chemical or nuclear reactions
– simulating materials or other popularized use cases (cryptography & encryption, etc)