first thing i looked for in the infiniattention paper. theres limits ofc but they do pretty well for the size
MACHINE LEARNING
-
Easy-to-use platform with hosted datasets and pretrained models
By
–
hey you said it! they make getting started really easy 🙂 and their hosted resources are so nice, like datasets & pretrained models
-
Google’s Linear Attention Breakthrough: Griffin and Infini-Attention
By
–
with Griffin and Infini-attention, it increasingly feels like Google leapfrogged Together and RWKV in the race for scaling up linear attention, and they shared a watercooler conversation with Anthropic or something papers are trickling out now but it's hard to know what's
-
HuggingFace Trainer DistributedDataParallel Gradient Bug Fix
By
–
ok so, it was a gradient bug. let me explain with the huggingface trainer, when training on multiple GPUs, sometimes the model is stored as an nn.Module, but sometimes it’s wrapped in this DistributedDataParallel thing I was inadvertently forwarding on the nn.Module in
-
DeepSpeed optimization successfully implemented for AI training
By
–
@winglian wrangled deepspeed till it worked
-
Snorkel Tops AlpacaEval: Training Approach and Metrics Changes
By
–
Hoang Tran, one of the researchers behind Snorkel's AlpacaEval Leaderboard-topping model, explains his training approach and why he asked for the leaderboard to change its metrics.https://t.co/xXve73ofo2 pic.twitter.com/YnSsbvZrfo
— Snorkel AI (@SnorkelAI) 11 avril 2024Hoang Tran, one of the researchers behind Snorkel's AlpacaEval Leaderboard-topping model, explains his training approach and why he asked for the leaderboard to change its metrics. https://
snorkel.ai/how-snorkel-to
pped-the-alpacaeval-leaderboard-and-why-we-re-not-there-anymore/
… -
AI Vision Models: Beyond Human Performance and Visual Pattern Recognition
By
–
This paper is about how AI vision models don't have human level performance, but also this: GPT-4 without any vision capabilities at all still does remarkably well just faking it, suggesting that, much like with words, there is more regularity in the visual world then we realize
-
Training Single Models with Cross-Entropy Loss
By
–
thanks for sharing — seems like this is some odd student-teacher setting though; i'm just training a single model with cross-entropy
-
Contrastive Learning with Large Batches for Embedding Models
By
–
sort of; i'm doing contrastive learning with large batches; at the end it's computed as cross-entropy, but this isn't a typical language modeling setup, i'm training an embedding model
-
Cosine Decay Implementation Bug in Training Loop
By
–
it's supposed to be cosine decay but this is just linear warmup and my thing is already broken