I think there's a lot we can learn from IBM Watson, good and bad.
@fchollet
-
IBM Watson and the 2014 AI Job Displacement Panic
By
–
The first "AI is coming for you job" mass panic I experienced first-hand was in 2014 and was based on the same premise — that human doctors would soon be a thing of the past. It was triggered by the now-defunct IBM Watson.
-
Deep Models Fail to Generalize on Puzzles Beyond Training Data
By
–
See e.g. https://
arxiv.org/abs/2212.09993 "Our experiments reveal that while powerful deep models offer reasonable performances on puzzles that they are trained on, they are not better than random accuracy when analyzed for generalization." -
Testing AI Performance: Familiarity vs. Novel Questions
By
–
The system is out there. You can just interact with it and test the hypothesis by yourself! You see an immediate correlation between question familiarity & performance, and a fast break down of performance as the questions become more novel — independent of question complexity!
-
GPT models fail at genuine generalization on novel problems
By
–
Exactly. Except with the opposite conclusion. Every study so far that tries to test GPT-N for actual generalization has found that it scores no better than random on genuinely new problems — brand new coding problems in particular. This is why it can't do ARC either.
-
Training Data Alone Doesn’t Ensure Quality AI Performance
By
–
Even a hashtable can pass the bar exam giving enough training data. But you probably don't want to be represented by a hashtable.
-
Skill Acquisition Efficiency as a True Measure of Intelligence
By
–
Skill, on its own, is not a sign of intelligence. But skill acquisition efficiency over arbitrary skillsets is. And that looks very different in the context of human test takers vs. machines.
-
General Intelligence vs Specialized Machine Design in Chess
By
–
And so they could have used these 30 years to become great at *any* other human skill. They possess general intelligence. But you cannot make the same intelligence assumption when you see a machine *designed* to play chess, or one that has memorized one billion chess games.
-
Invitation to discuss perspectives on large language models
By
–
Gary, I know you enjoy disagreeing, but I think you'd like some of my takes on LLMs
-
Notebook with strong validation accuracy but zero leaderboard score
By
–
Here's another notebook, much simpler than the last. Decent statistical power on the validation data (70% val acc vs 56% for the baseline) but scores 0 on the leaderboard The previous notebook managed a non-zero LB score at least (ranked #7 att) https://
kaggle.com/fchollet/per-p
ixel-classification-2-gpus/edit
…