there are some other nice properties: – we can get good performance with very little data
– we recover embeddings that "look" like the original dataset
– when some percentage of the original data is available, we generally select that data first
@jxmnop
-
Embeddings Recovery with Minimal Data Requirements
By
–
-
SELECT: Training Models with Limited Data Access
By
–
…but it works we try a bunch of baselines like selecting data based on likelihood or even the top-k per-class our method (SELECT) can train a model to similar performance, with access only to model weights and some web data
-
Scaling Per-Example Gradients: Algorithm Efficiency Improvements
By
–
our algorithm is a bit complicated, mostly because computing per-example gradients is hard to do at scale so we make some efficiency improvements:
– computing grads w vmap
– only using last-layer grads (which are still big, in the case of LMs)
– projecting them to a smaller dim -
Finetuning AI Models: Gradient-Based Data Selection Strategy
By
–
to do this, you need TWO sets of model weights: the initial model and a finetune this is realistic. open-weights models often come with two checkpoints instead of one-shot generating data from weights, we select data from the web with gradients that point along the model diff
-
Reverse Engineering Training Data From Language Model Weights
By
–
NEW RESEARCH: Approximating Language Model Training Data from Weights ever wonder how much information is available in an open-weights model? DeepSeek R1 weights are 1.2 TB… what can we learn from all those bits? our method reverses LLM finetuning to recover data:
-
LLMs Saved AI From Surveillance and Ad Targeting
By
–
people don't realize it, but LLMs saved AI if we didn't stumble into general intelligence most people would still be working on vision, which was most useful for human surveillance and ad targeting.. inherently evil technology
-
ASI Scientific Foundations: Beyond LLMs and Learning Algorithms
By
–
part of an ongoing effort to think about what ASI could actually mean from a scientific perspective. > is superintelligence just some form of LLMs?
> what is the learning algorithm? > what is the data? read more here: -
Superintelligence: Supervised or Reinforcement Learning Path
By
–
# Superintelligence from first principles some have said that superintelligence is "an engineering problem"; that we know how to build it now. but there are only two ways to learn: supervised and reinforcement learning. which one will get us there? i wrote about this:
-
vLLM Prefix Caching: Optimization Technique Explained
By
–
here's my favorite, the explanation of vLLM prefix caching: http://
docs.vllm.ai/en/v0.8.5/desi
gn/automatic_prefix_caching.html
… -
Researcher offers arXiv endorsement for good faith AI research
By
–
if you did good-faith research and the endorsement is your only barrier to putting it on arXiv, send it to me and i will endorse you