Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries in 2021 these folks measure the exact number of 'statements' models can memorize (a) capacity scales linearly with parameters
(b) more training samples -> less memorization
RESEARCH
-

Language Models Knowledge Storage Capacity and Memorization Scaling
By
–
-

RNN Capacity: 5 Bits Per Parameter Storage Empirical Analysis
By
–
Capacity and Trainability in Recurrent Neural Networks this very nice paper from 2016 (!) estimates certain types of RNNs can store 5 bits/param. they find this empirically and measure bits using the entropy of the distribution of class labels
-

Neural Network Memorization Capacity Under Parameter Quantization
By
–
Memorization Capacity of Deep Neural Networks under Parameter Quantization in 2019 these folks measured the capacity of RNNs/CNNs/MLPs, also using this class-based entropy idea they quantized the networks to half-precision and saw a capacity drop (but not nearly by half)
-

Single-Layer Perceptron Memory Capacity Foundational Result
By
–
A foundational result in machine learning is that a single-layer perceptron with N^2 parameters can store at least 2 bits of information per parameter (Cover, 1965; Gardner, 1988; Baldi & Venkates) specifically: an MLP can memorize 2 N^2 things with only N^2 parameters
-

Capacity Measurements for Language Models Research
By
–
in prep for our new research dropping on ArXiv tomorrow (i think), here is a thread about…. CAPACITY MEASUREMENTS FOR LANGUAGE MODELS
-

Meta-Reinforcement Learning Tutorial: A Comprehensive Guide
By
–
A Tutorial on Meta-Reinforcement Learning Beck et al.: https://
arxiv.org/abs/2301.08028 #ArtificialIntelligence #MetaLearning #ReinforcementLearning -
LLM Coding Capabilities: From GPT-2 to GPT-4o Progress
By
–
> 2020: gpt2 can't write code
> 2021: gpt3 can't reliably write python
> 2022: instructgpt can't write blocks of code without syntax errors
> 2023: chatgpt can't do leetcode
> 2024: gpt4 can't debug CUDA
> 2025: gpt4o can't implement entire PR these LLMs are not to be trusted… -

Year of the Agent: Critical Analysis of AI Advancement Limits
By
–
“2025 is the Year of the Agent” — everyone. But once you sweep away the hype, what actually changed? Here’s the condensed field report—from someone working in the field much before any "LLM" came to exist: 1. We (potentially) hit the prediction ceiling. Pre-training gains
-
Google Launches AlphaEvolve Early Access Program with Academic Partners
By
–
What’s next? Google is preparing an early access program for AlphaEvolve with academic partners. And yes a friendly UI is coming soon. AlphaEvolve may eventually expand into materials science, biotech, and more.
-

AlphaEvolve: LLMs and Evaluators Discover New Algorithms
By
–
Why AlphaEvolve matters This isn’t just about math. AlphaEvolve shows: – LLMs + evaluators can discover new knowledge
– Code generation can evolve into true algorithm innovation
– AI agents can now tackle hard science problems autonomously