Introducing MultiChallenge by @scale_AI – a new multi-turn conversation benchmark. Current frontier LLMs score under 50% accuracy (top: 44.93%). o1
Claude 3.5 Sonnet
Gemini 2.0 Pro Experimental Paper: http://
arxiv.org/abs/2501.17399
Leaderboard: http://
scale.com/leaderboard/mu
ltichallenge
…
@alexandr_wang
-
MultiChallenge: New Multi-Turn LLM Benchmark Released by Scale AI
By
–
-
OpenAI Achieves 26.6 Score on HLE with Deep Research
By
–
Congrats to OpenAI on getting 26.6 on HLE with deep research! We look forward to evaling OpenAI deep research on the Humanity Last Exam private set soon!
-
Chinese Government Data Labeling Subsidies and Support Programs 2023
By
–
3/ In 2023, the Chinese Government announced data labeling subsidies including:
– tax incentives
– government procurement of data
– data vouchers for cost reduction
– investment promotion
– talent development programs -
China integrates 81 LLM contracts in military and government sectors
By
–
2/ in the first half of 2024, the Chinese government issued 81 contracts to LLM companies to integrate it into their military and government, with Zhipu emerging as the early winner
-
Bank of China Announces $140B AI Supply Chain Investment
By
–
1/ The Bank of China announced 1 trillion yuan ($140B) in investments for the AI supply chain on Tuesday, Jan 28, 2025 in response to Stargate.
-
China’s Serious Commitment to AI Dominance: Three Key Evidence Points
By
–
Three data points on how serious China is about winning in AI:
-
Prompts and answers as human data for synthetic chain-of-thought generation
By
–
prompts (and answers) are the human data. that enables you to generate the synthetic CoT
-
DeepSeek R1 and v3: Impact on LLM Training Data
By
–
What does DeepSeek R1 & v3 mean for LLM data? Contrary to some lazy takes I’ve seen, DeepSeek R1 was trained on a shit ton of human-generated data—in fact, the DeepSeek models are setting records for the disclosed amount of post-training data for open-source models: – 600,000
-
Trump calls DeepSeek release wake-up call for US industry competition
By
–
"The release of DeepSeek AI from a Chinese company should be a wake up call for our industries that we need to be laser-focused on competing to win." — President Trump https://t.co/LhAj1p3J4V
— Alexandr Wang (@alexandr_wang) 28 janvier 2025"The release of DeepSeek AI from a Chinese company should be a wake up call for our industries that we need to be laser-focused on competing to win." — President Trump
-
DeepSeek Wake-Up Call: America Must Innovate Faster and Control Chip Exports
By
–
DeepSeek is a wake up call for America, but it doesn’t change the strategy: – USA must out-innovate &race faster, as we have done in the entire history of AI
– Tighten export controls on chips so that we can maintain future leads Every major breakthrough in AI has been American