clearly in five years SOTA AI models will train on a single string containing approximately 2^21 tokens
@jxmnop
-
Training Methods Critical for AI Model Performance
By
–
well they have to train it like this or it wouldn't work nearly as well
-
AI Datasets Shift: Fewer Examples, Longer Sequences
By
–
an interesting trend in AI is that the best datasets have fewer and fewer longer and longer sequences dataset five years ago:
~10^5 examples, each of 2^6 tokens nowadays:
~10^3 examples, each of 2^15 tokens it’s actually more data. but the tokens are stacked horizontally now -
Where Are Encoder-Decoders Being Used in AI?
By
–
oh where have you been seeing encoder-decoders come up?
-
RETRO: Revisiting DeepMind’s Knowledge Outsourcing Architecture
By
–
RETRO (DeepMind, 2021) is a beautiful idea, one badly in need of revisiting the central innovation of retro is to have a small model decide what token to predict next, but outsource all knowledge to a large offline datastore this has the added benefit of allowing you to insert
-
Open Models Cannot Match This Capability Yet
By
–
we don't have any open model that produces anything remotely like this and that is my point
-
DeepSeek outputs strike different compared to other AI models
By
–
yes me too! but it's still so different than the deepseek outputs, so striking
-
OpenAI vs DeepSeek: AI Reasoning Race Accelerates Dramatically
By
–
these reasoning traces have been keeping me up at night on the left: new OpenAI model that got IMO gold
on the right: DeepSeek R1 on a random math problem you need to realize that since last year academia has produced over a THOUSAND papers on reasoning (probably much more). -
Modern Medicine Advances and Healthcare Innovation
By
–
yeah, every day it’s not that bad. modern medicine is incredible
-
Thyroid Cancer Diagnosis Led Me to ML and NLP
By
–
in college i was diagnosed with an unusual thyroid cancer. i tried building a pipeline to aggregate & summarize information from pubmed and clinical trials to my great surprise the summarization part was pretty bad. this got me interested in ML and NLP