word2vec didn't do spatial IIRC.
@jeremyphoward
-
TPUs Integration Challenges with PyTorch Framework
By
–
TPUs are still a bit of a pain (to say the least!) to get working with PyTorch, and PyTorch is the easiest way still to get stuff done, on the whole.
-
Spatial Entity Embeddings: Historical Context Since 2017
By
–
The spatial locations thing has been around since at least the entity embeddings paper. IIRC that was around 2017 or so.
-
PEFT LoRA Implementation Guide for Model Fine-tuning
By
–
Just paste it directly above your call to get_peft_model(). There's many examples around, eg: https://
github.com/huggingface/pe
ft/blob/main/examples/conditional_generation/peft_lora_seq2seq.ipynb
… -
BNB Quantized Layers Initialization Performance Issues
By
–
The issue is the creation of the bnb quantised layers, not the adapters. (They are fast to init anyway since they're small.)
-
WhatsApp’s Engineering Challenges: Lessons Beyond ML Data Engineering
By
–
Consider what Whatsapp achieved prior to acquisition with their team. And their eng challenges were many orders of magnitude harder than ML data eng!
-
RLHF Training with Small Teams and Data Engineering
By
–
I've seen plenty of folks doing RLHF with <5 dedicated people! (And I do all my own data engineering — it doesn't take too long after a few decades of practice…)
-
Model Sharding Challenges for Large GPU Deployment
By
–
Not sure if there's a way to do that with model sharding — which is necessary if your model is too big to fit on the GPU.
-
Hugging Face Performance Fix Coming Soon Official Update
By
–
I chatted to the Hugging Face folks and they tell me an official fix for this performance issue is on the way! 😀 So hopefully my hacky workaround will be redundant soon…
-
PEFT Kaiming Init Optimization Patch for Training
By
–
The issue is that PEFT spends all that time doing unnecessarily doing kaiming init. So our code first patches the `init` module so that `kaiming_uniform_` does nothing at all. Here's a gist with the full code: