MACHINE LEARNING

Deep Models Fail to Generalize on Puzzles Beyond Training Data

By

–

07 April 2023 23h26

See e.g. https://
arxiv.org/abs/2212.09993 "Our experiments reveal that while powerful deep models offer reasonable performances on puzzles that they are trained on, they are not better than random accuracy when analyzed for generalization."

Testing AI Performance: Familiarity vs. Novel Questions

By

–

07 April 2023 23h25

The system is out there. You can just interact with it and test the hypothesis by yourself! You see an immediate correlation between question familiarity & performance, and a fast break down of performance as the questions become more novel — independent of question complexity!

GPT models fail at genuine generalization on novel problems

By

–

07 April 2023 23h23

Exactly. Except with the opposite conclusion. Every study so far that tries to test GPT-N for actual generalization has found that it scores no better than random on genuinely new problems — brand new coding problems in particular. This is why it can't do ARC either.

Training Data Alone Doesn’t Ensure Quality AI Performance

By

–

07 April 2023 22h39

Even a hashtable can pass the bar exam giving enough training data. But you probably don't want to be represented by a hashtable.

Skill Acquisition Efficiency as a True Measure of Intelligence

By

–

07 April 2023 22h28

Skill, on its own, is not a sign of intelligence. But skill acquisition efficiency over arbitrary skillsets is. And that looks very different in the context of human test takers vs. machines.

General Intelligence vs Specialized Machine Design in Chess

By

–

07 April 2023 22h27

And so they could have used these 30 years to become great at *any* other human skill. They possess general intelligence. But you cannot make the same intelligence assumption when you see a machine *designed* to play chess, or one that has memorized one billion chess games.

→ View original post on X — @id_aa_carmack

Language Models Trained on Vastly More Data Than Humans Encounter

By

@id_aa_carmack

–

07 April 2023 20h02

A person is exposed to on the order of a billion words over their entire life through speech and reading. Modern large language models are fed hundreds of times that in the weeks of their training.