by the way, seems like as good a time as any to let the internet know I’ll be interning at FAIR next summer! excited to meet new people, get a temporary change of scenery, and use the GPUs and if you’re in SF, hit me up 🙂
@jxmnop
-
HuggingFace Trainer DistributedDataParallel Gradient Bug Fix
By
–
ok so, it was a gradient bug. let me explain with the huggingface trainer, when training on multiple GPUs, sometimes the model is stored as an nn.Module, but sometimes it’s wrapped in this DistributedDataParallel thing I was inadvertently forwarding on the nn.Module in
-
Training Single Models with Cross-Entropy Loss
By
–
thanks for sharing — seems like this is some odd student-teacher setting though; i'm just training a single model with cross-entropy
-
Contrastive Learning with Large Batches for Embedding Models
By
–
sort of; i'm doing contrastive learning with large batches; at the end it's computed as cross-entropy, but this isn't a typical language modeling setup, i'm training an embedding model
-
Cosine Decay Implementation Bug in Training Loop
By
–
it's supposed to be cosine decay but this is just linear warmup and my thing is already broken
-
Model Degradation Without Gradient Norm Increase Troubleshooting
By
–
model-trainers of Twitter: why does my model start getting worse over time without the grad norm noticeably increasing — how can this be possible? and how do i fix it? (again, please help 🙂 )
-
Discrete Space Prompting vs Prefix Tuning Model Optimization
By
–
well i'm pretty certain it all happens in *discrete* space, via prompting, and strings; prefix tuning requires training a model, taking gradients, dense parameters, etc.
-
Overcoming Research Blocks: Strategies for Problem Solving
By
–
a common shared experience of the research process seems to be "getting stuck" what am i supposed to do when i'm stuck on a problem? read papers? write unit tests? meditate, go for a walk? or should i just think harder? (please help)
-
GPU Computing: Mining Crypto versus Building AI Intelligence
By
–
why waste your GPUs mining bitcoin when you can literally turn your compute into intelligence
-
Synthetic Data Pre-caching Issues in ML Architecture
By
–
i think it is a data issue, since synthetic data induces pre-caching with the same arch