class tonight. send me your most scrutable visualizations of Transformers
@jxmnop
-
Essential ML Profiler: CPU GPU Code Analysis Tool
By
–
the most-needed ML startup: a really good profiler for CPU and GPU code (a "tensorboard for profiling") something that's:
– trivial to set up
– fast to run & analyze
– integrates with frameworks
– makes it easy to see what's happening on CPU and GPU
– suggests solutions for you -
Text versus 32-bit floats: lossless compression efficiency trade-offs
By
–
I don’t think that’s necessarily true! text is actually relatively few bytes, while each 32-bit float is 4 bytes, so at least at short lengths things could be lossless
-
Text Embeddings Lossy Collisions Detection Problem
By
–
yeah! but I don’t know if text embeddings are necessarily lossy? and if so how to locate collisions? (but yes; this is the entire point)
-
Model Collisions: When X and Y Have Different Meanings
By
–
oh also I think what would be really interesting is collisions where x and y do mean different things In this case it’s pretty intuitive; 1700 purples “means” essentially the same thing as 1702 purples; the model is working as expected next up is (x, y) with different meanings
-
TSNE Visualization of Purple String Embeddings Structure
By
–
BONUS TSNE: here's a graph of the embeddings of the different-lengthed purple strings, colored by the number of purples what a strange structure!
-
Text Embedding Collision Discovery: Purple Repetition Experiment
By
–
I think I found the first evidence of a collision in text embeddings! i.e. for an embedding function f, i found a text pair x, y where x ≠ y but emb(x) = emb(y) ok granted it isn't thatttt interesting but what I did was repeat the word purple from 1 to 8191 times. 'purple ' is
-
Mathematician’s Approach to Language Model Research and Theory
By
–
visiting some mathematician friends who work on language models and i'm struck by how different the lifestyle is. their workspaces are mountains of papers surrounded by endless blackboards. they talk of mutual info, topological theory, PAC bounds. i am completely covered in chalk
-
Adafactor vs Adam: Optimizer Choice for Small Neural Networks
By
–
ok good point and i AGREE that the details matter but these kids aren't trying to replicate T5 pretraining in 20 hours on a single gpu; they just want to train little neural networks from scratch; and for their use cases adafactor vs adam won't almost ever matter
