Regulation starts at roughly two orders of magnitude larger than a ~70B Transformer trained on 2T tokens — which is ~5e24. Note: increasing the size of the dataset OR the size of the transformer increases training flops. The (rumored) size of GPT-4 is regulated.
@soumithchintala
-
GPT-4 Training Data Transparency: YouTube and Propaganda Sources
By
–
list out all the user data and internet data the model was trained on. Is GPT4 trained on Youtube? On propaganda videos specifically? That's the kind of transparency that will be strictly useful
-
AI Executive Order Drives Transparency and Government Regulation
By
–
The AI Executive Order is here, for better or worse.
(my thoughts further below) It directs the government to move AI towards transparency, adding bureaucracy, process and standards, increasing R&D funding and directing fairly sweeping policy changes across a wide range of -
Search LLM API Integration: Current Availability and Options
By
–
Is there a Google/Bing Search + LLM available as an API today to try? * @perplexity_ai is the best here, but their search+LLM not yet available as an API
* Bard APIs are not openly available yet (need to do some data processing asap) -
China AI chip sanctions may fork AI technique development
By
–
Currently, we overfit to exploring AI techniques that work well on NVIDIA GPUs.
With the AI Accelerator sanctions on China — one interesting result might be that China forks silicon and explores a different idea-space in AI techniques than the rest of the world. -
PyTorch Conference Attendance Highlights Community Engagement
By
–
thanks for coming to your first pytorch conference. it was great having you there!
-
Float8 Training Implementation with Stateful Tensor Subclass
By
–
my bad, didn't realize it was not public yet. We'll work on publishing it asap. We basically have a stateful Tensor subclass and a Float8Linear layer that work together to get to working Float8 training.
-
PyTorch Float8 Experimental Documentation Released
By
–
this has a lot more documentation https://
github.com/pytorch-labs/f
loat8_experimental
… -
PyTorch Conference 2026: Major Announcements on Performance
By
–
pretty excited for the @pytorch conference today and tomorrow. Some very interesting work being talked about in our Maintainers Keynote in the morning…a big one and a few smaller-but-super-high-impact ones (my favorite being around perf). hope to catch you tomorrow!!!
-
Open Licensing Clause for LLMs: GPL Equivalent Model
By
–
In Open Licensing, what is the "Linking and Derived Works Clause" equivalent for LLMs that will help them progress further like it did for GPL and GNU/Linux?