maybe we just haven't found the right form factor
@jxmnop
-
Understanding Transformers via N-Gram Statistics: Physics for Language Models
By
–
the closest thing i've seen to actual "physics for LMs" was probably this (single-author!) paper from neurips 2024: Understanding Transformers via N-Gram Statistics this is how we used to think about LMs; not sure why we stopped.
-
New Datasets Continue to Drive AI Development
By
–
luckily, they’re not running out of new datasets:
-
AI Model Scaling Limits: Tech Companies Face Innovation Challenges
By
–
have no idea about meta specifically, but the same thing is happening at ALL tech companies they’re running out of ideas to improve the models, can’t 10x the amount of text written by humans, can’t afford to 10x training compute again this next year is gonna be interesting
-
Big Tech Companies Developing Digital Nose Technology
By
–
ive heard something very interesting through the AI grapevine; apparently, lurking deep in the shadows of the big tech companies, there lies a group of smart and well-funded AI researchers building a Digital Nose these scientists, so far, have only succeeded in digitizing and
-
Why Deep Derivatives Knowledge Isn’t Critical for Machine Learning
By
–
i guess it turns out that understanding derivatives deeply doesn't help you that much in ML
-
From Manual Calculus to Automatic Differentiation in ML
By
–
another funny thing about the state of ML research in 2015 was that many people spent much of their time computing derivatives by hand. no, really. this was before autodiff, and half the battle was doing basic calculus without making mistakes now you just type .backward()
-
The Golden Age of AI Research: 2015 Optimism and Fundamentals
By
–
it must have been more fun to be an grad student in AI and ML circa 2015, when there was an immense amount of optimism that if you understood something True about the world (e.g. fundamentals of lingustics or the human visual cortex) then you could use that knowledge to build
-
Reasoning tokens alone cannot solve fundamental AI limitations
By
–
Adding more reasoning tokens is not going to fix this