See e.g. https://
arxiv.org/abs/2212.09993 "Our experiments reveal that while powerful deep models offer reasonable performances on puzzles that they are trained on, they are not better than random accuracy when analyzed for generalization."
@fchollet
-
Deep Models Fail to Generalize on Puzzles Beyond Training Data
By
–
-
Testing AI Performance: Familiarity vs. Novel Questions
By
–
The system is out there. You can just interact with it and test the hypothesis by yourself! You see an immediate correlation between question familiarity & performance, and a fast break down of performance as the questions become more novel — independent of question complexity!
-
GPT models fail at genuine generalization on novel problems
By
–
Exactly. Except with the opposite conclusion. Every study so far that tries to test GPT-N for actual generalization has found that it scores no better than random on genuinely new problems — brand new coding problems in particular. This is why it can't do ARC either.
-
Training Data Alone Doesn’t Ensure Quality AI Performance
By
–
Even a hashtable can pass the bar exam giving enough training data. But you probably don't want to be represented by a hashtable.
-
Skill Acquisition Efficiency as a True Measure of Intelligence
By
–
Skill, on its own, is not a sign of intelligence. But skill acquisition efficiency over arbitrary skillsets is. And that looks very different in the context of human test takers vs. machines.
-
General Intelligence vs Specialized Machine Design in Chess
By
–
And so they could have used these 30 years to become great at *any* other human skill. They possess general intelligence. But you cannot make the same intelligence assumption when you see a machine *designed* to play chess, or one that has memorized one billion chess games.
-
Invitation to discuss perspectives on large language models
By
–
Gary, I know you enjoy disagreeing, but I think you'd like some of my takes on LLMs
-
Notebook with strong validation accuracy but zero leaderboard score
By
–
Here's another notebook, much simpler than the last. Decent statistical power on the validation data (70% val acc vs 56% for the baseline) but scores 0 on the leaderboard The previous notebook managed a non-zero LB score at least (ranked #7 att) https://
kaggle.com/fchollet/per-p
ixel-classification-2-gpus/edit
… -
AI Sensitivity to Flattery as a Security Vulnerability
By
–
It is very sensitive to flattery — that's the backdoor
-
Progress follows sigmoid curves, appearing linear from distance
By
–
Progress is neither linear nor exponential, it's a series of sigmoids (which ends up looking linear when you zoom out)