Benchmarks should have expiration dates.
MACHINE LEARNING
-
ML Demo Hype: Why Cherry-Picked Examples Fool Investors
By
–
Baffled by the number of people (specifically influencers and investors) who still get fooled by ML demos and cherry picked examples engineered for wow effect! Rookie mistake to judge the quality of a model (versus looking at usage, revenue and more tangible evaluations)
-
GPT-3 cannot perform accurate calculations without step-by-step writing.
By
–
In general, you shouldn't expect it to be able to perform accurate calculations, at least not "in its head" — this is a known limitation of models like GPT-3. It only stands a chance if prompted to write calculations out step-by-step like one might do on paper.
-
ChatGPT avoids hallucination on Hofstadter/Bender questions
By
–
You're talking about a different model — this post is about ChatGPT. text‑davinci‑003 still fails on all of the Hofstadter/Bender questions that I've tried. The prompt you're suggesting does not seem to produce a hallucination in ChatGPT:
-
Challenge of incorporating all known trivia into training data
By
–
Expecting it to fully integrate every piece of trivia seems unfair — a lot of people would rate the first answer as reasonable. It’s fundamentally difficult to get training data that incorporates everything the model knows, rather than what the human demonstrator knows.
-
GPT-3 improvement shows not just pattern-matching, still hallucinates details
By
–
The point isn’t that it’s perfect, just that it’s not narrowly pattern-matching on this specific list of questions — there’s clear improvement vs. GPT-3 across many questions that contain false assumptions. It does still hallucinate details in other ways.
-
Tokenization issue: model struggles with letter sequences
By
–
This feels like a tokenization issue — in general, it doesn’t see text as sequences of letters, and struggles with tasks that assume it does.
-
Reinforcement Learning as Backpropagation Through Time
By
–
Reinforcement learning is backpropagation through time.
-
GPT-3 succeeds on trick questions, pre-training unchanged since 2021
By
–
No. It also succeeds on questions I’ve written myself that trick GPT-3, as in this screenshot. The pre-training data hasn’t been updated since 2021, and OpenAI specifically denies (in this thread) training on the Hofstadter/Bender questions.
-
Domino MLOps Platform: Hybrid Multi-Cloud Data Science Solution
By
–
What did the truck say? Domino is the only hybrid and multi-cloud MLOps platform. Unleash data science, not cloud spend. Follow us to #ReInvent2022 booth #2938 for more info, and to win prizes! https://t.co/I7YNKEJaZT pic.twitter.com/5oNDRyWaJG
— Domino Data Lab (@DominoDataLab) 30 novembre 2022What did the truck say? Domino is the only hybrid and multi-cloud MLOps platform. Unleash data science, not cloud spend. Follow us to #ReInvent2022 booth #2938 for more info, and to win prizes! https://
domino.buzz/3hOw8Ip