see this link below, it was a failure and we stopped development. @zou3519 can give any more insights (or @srush_nlp )
naming dimensions is cool until you have to preserve the names across shape propagation (the output of an operation who's input is a named tensor).
so named
@soumithchintala
-
Named Tensors in Deep Learning: Shape Propagation Challenges
By
–
-
torch.export maturing for inference graphs in PyTorch
By
–
it comes with its own temporary friction, torch.export is still maturing for backward/training graphs and all effort over the past two years was put on inference maturity
-
torch.export: Compile-time autotuning for training without runtime JIT
By
–
if you want to avoid guards and do autotuning at compile-time, have no jit at runtime, consider torch.export; thats what you'd want.
in fact, @ezyang has been pushing it for some training workloads that dont want any jitting at runtime and ones that need a bunch of sharding info -
JAX adoption challenges: GPU and TPU experience insights
By
–
the jax topic didn't even come to my table (I'm also out on leave). afaik the biggest pushback to using jax came from msl folks who had to use jax+gpus in their previous place of employment and didn't think it was great on a few important dimensions.
the jax+tpus experience -
Meta’s Open AI Infrastructure: Industry’s Hidden Messiness
By
–
AI infra is young, hard and messy (you obviously know this). the fact that meta talks about their infra details openly only makes it publicly messy. everyone's infra across the industry is pretty messy but privately so.
-
PyTorch Fork Infrastructure: Low Precision Types at Scale
By
–
from what i know (folks who worked on the infra bits at the other lab), the other lab forked pytorch several years ago for their large-scale runs before tensor subclasses were a thing, as they needed low and mixed precision types. now they have so much custom stuff on top of
-
PyTorch Compilation Strategy for Large-Scale Transformer Models
By
–
1. there is no fork of pytorch fwiw. only a fork of nccl. clearly you overheard wrong.
2. torch.compile wasn't even considered because it doesnt have precompilation yet and that was deemed a requirement for large-scale xlformers runs. so the whole thing about graph breaks is -
ChatGPT’s Em Dashes Made AI Prose Unmistakably Soulless
By
–
I used em dashes (a lot) before ChatGPT made them the official punctuation of soulless AI prose. (can't believe I've had to gentrify my own writing style)
-
From VFX Dreams to AI Research: A Higher-Order Attack
By
–
Tried being a VFX artist for films. Flopped.
So, got into vision/ML research to build agents that did what I couldn’t.
Higher-order attack on the same dream. (eventually broadened my interest within vision/ML research, but that's how I got into AI research) -
TorchTitan Model Configuration and Zero3 Zero2 Trade-offs
By
–
thanks for sharing the configs.
The model is too small for the defaults to be tuned around it. ( thanks @lessw2020 for looking into it).
Most folks who use TorchTitan don't quite train in that distribution.
Also, the zero3 -> zero2 is a compute/memory trade-off).