an unsung hero of the Transformer's success is not the attention mechanism per se, but *causal masking* it let us simplify two for loops to just one: > RNN
for t in [1..S]: for i in [1..L]: x[t], h = layers[i](x[t], h) > transformer for i in [1..L]: x = layers[i](x)
@jxmnop
-
Causal Masking: The Unsung Hero Behind Transformer Success
By
–
-
One Code Line Reduces PyTorch Downloads by 400MB
By
–
one line of code just reduced pytorch download sizes by 400MB . for literally everyone lol so much low hanging fruit when you pay this much attention to detail
-
Deep Learning Curriculum Standard in Advanced AI Education
By
–
pretty much every deep learning class includes a few weeks on it
-
Reinforcement Learning: Neglected Tool in Language Models
By
–
we all learn reinforcement learning in intro ML. it’s widely accepted as foundational, a pillar of the field yet outside of RLHF, everyone working on language models ignored RL for years folk wisdom said it “doesn’t work” what other fundamental tools are we neglecting?
-
Man Tells Mother Stop Saving Retirement Due AGI Risk
By
–
remember that one guy who said he told his mom to stop saving for retirement because of AGI
-
H200 Hardware Achieves 10k Tokens Per Second Performance
By
–
it's an h200! i'm pretty sure you can get up to 10k tokens/second with some configurations
-
Distillation increases model capacity efficiency
By
–
it seems pretty likely distillation would increase capacity if that’s what u mean
-
Compression: Key to AI Model Efficiency and Performance
By
–
i think the answer has something to do with compression btw