overall a really entertaining read, highly recommend. here's a link http://
jmlr.org/papers/volume3
/bengio03a/bengio03a.pdf
…
@jxmnop
-
Foundational Machine Learning Research Paper Highly Recommended
By
–
-
Neural Networks Outperform N-gram Models in Mixture Architecture
By
–
their neural networks worked better than any other methods and the bigger ones are better although at this scale, mixing with ngrams still helps a lot. they use a "mixture of models" – similar to today's MoEs, but the experts are different ngram models, plus one neural network
-
CPU vs GPU: Computing Power Gap in AI Training
By
–
they trained on a cluster of 64 1.2 GHz CPUs i did some math and think this is roughly 1 teraflop. a single H100 GPU gives you 4,000 teraflops lol
-
Model Training Scale: From 800K to 25 Trillion Tokens
By
–
their model is trained on 800K words, which is around 2.5M tokens. remember today's models are up to ~25T tokens (10^7 more, or 10 million million times) (they trained another model on 15M words)
-
Compact Language Model Architecture: 12M Parameters vs LLAMA 4
By
–
they use word embeddings + single-layer MLPs vocab size is 18K, context window is 6 words, hidden dimension is 60, word embedding have 100 dimensions their model has approx |V|(nm + h) = 17,964 × (6 × 100 + 60) = 12 million parameters about 200K times smaller than LLAMA 4…
-
Hogwild Parallel Processing on CPU: Engineering Techniques
By
–
they do parallel processing (on CPU). i think this is now a well-known technique called Hogwild that wasn't formally developed until 2011. engineering this sounds pretty fun:
-
Neural Networks Training History: The 2003 Montreal Paper
By
–
did you know people have been training neural networks on text since 2003? everyone talks about Attention Is All You Need. but this is the real paper that got our field started. it was in 2003, in montreal. i read it, and it was even more forward-thinking than i expected:
-
Smart Actors Adapt Strategy or Hide Knowledge
By
–
i believe you, but they are also very smart, and likely to change course in the face of evidence. so i'd guess either (a) they will start trying something new or (b) they know something that we don't
-
SAEs criticism fades as interpretability researchers pivot forward
By
–
i don't see value in repeatedly dunking on SAEs, esp after google stated this publicly. interpretability people are already looking for a new paradigm. "The pessimist complains about wind; the optimist expects it to change; the realist adjusts the sails." – William Arthur Ward
