it might take years, but I think it's already pretty popular (in my ML circles at least).
@rasbt
-
TorchTune Library Growth Inspired by LitGPT
By
–
Oh yeah, torchtune is a good library. It was modeled after our LitGPT library and has grown quite a bit!
-
DevContainers Setup Guide and Local Machine Execution Preferences
By
–
There's a setup guide for DevContainers here, too: https://
github.com/rasbt/LLMs-fro
m-scratch/tree/main/setup/03_optional-docker-environment
…
No offense, but personally I had running it on my local machine. It has slower startup times, adds overhead, and is also storage-inefficient. -
Comparing Energy Consumption: Video Streaming vs AI Model Inference
By
–
Watching 20 min of a 4k video on YT is ~0.5 kWh Running 1000 Llama 3 8B prompts on a MacBook Pro: ~0.0025 kWh Running 1000 GPT 4o prompts: ?
-
Separate QKV Matrices Offer Better Big O Performance
By
–
Yes that's fair, in big O terms the separate QKV should be faster
-
GPU Precision Evolution: From 32-bit to FP8 in Transformers
By
–
Yes. Originally GPUs were designed for 32 bit precision, but in transformer contexts, you get away with even lower precision (16 bit is the standard usually). In recent months, people pushed that even further to FP8. One of the most recent architectures that successfully did FP8
-
Initial results show 32% higher costs with small context
By
–
that's true, based on the initial results, based on the small context size, it was already 32% more expensive.
-
Intellectual Curiosity Behind Parameter Reduction
By
–
just intellectual curiosity reducing the number of params
-
Simplifying Transformer Blocks: Key Research Paper
By
–
Yes. There was a nice paper on that, which looked into some other modifications: Simplifying Transformer Blocks, https://
arxiv.org/abs/2311.01906 -
Drop-in replacement code for modified attention mechanism available
By
–
Yup, the reader posted the code for a drop-in replacement for the modified attention mechanism. Just copy and paste it into the Chapter 5 code, which is available in the GitHub repo linked above.