*ai companies immediately overfit on the task of flying to boston*
@jxmnop
-
AI excels at math proofs but fails at booking travel
By
–
now AI can write novel proofs at the level of a world-class competitive mathematician but it still can’t reliably book me a weekend trip to boston so strange
-
Overfitting Criticism: The AI Model Performance Paradox
By
–
if they did well, you’d be complaining that they overfit
-
All AI Models Might Be The Same: Universal Semantics
By
–
new blog post "All AI Models Might Be The Same" in which i explain the Platonic Representation Hypothesis, the idea behind universal semantics, and we might use AI to understand whale speech and decrypt ancient texts
-
Memory-Native Language Models: The Missing Architecture Innovation
By
–
again, the AI labs are obsessed with building reasoning-native language models when they need to be building *memory-native* language models – this is possible (the techniques exist)
– no one has done it yet (no popular LLM has a built in memory module)
– door = wide open -
Training Models to Reconstruct Token Embeddings from Activations
By
–
i think i see your point. the embeddings we work with are the average of many tokens’ last-layer activations. so we can’t directly load them back in, need to train a model to do it
-
LLM Sequence Recovery Accuracy and Latent Space Configuration
By
–
it depends on sequence length, for long sentences we get about 94% back exactly. there are a lot of results in the paper (
http://
arxiv.org/abs/2310.06816) what does you mean 'set the latent space of the LLM'? -
Document Embeddings: Arithmetic and Text Decoding Techniques
By
–
I built this during my PhD! you can embed *documents* using the OpenAI embeddings, do arithmetic on them, and then decode them back to text e.g. E = emb(“Elon Musk bought Twitter in 2022”) – emb(“Elon Musk”)
vec2text(e) = “Someone bought twitter in 2022” -
AI Automation Eliminates Repetitive Code Work Forever
By
–
i feel intense pang of nostalgia thinking about how i will never have to write an immense amount of repetitive code to eg build a website ever again. there was a quiet bluecollar satisfaction to this type of work, knowing that if you don’t stay up all night smacking keys the