Any chance there'll be some info whether and how reasoning is implemented? E.g.,
(1) Inference-time scaling
(2) pure RL
(3) RL + SFT
(4) SFT/Distillation-based reasoning?
(5) 1 + 3?
@rasbt
-
Inquiry About Reasoning Implementation Methods in AI Models
By
–
-
Python package installation options with pip and uv
By
–
If they have Python installed separately, they can still do `pip install` or `uv pip install` etc, whatever works for them.
-
Conda speed advantage lost when using PyPI packages with pip
By
–
It's only fast if you use conda packages. But most AI/ML packages need to be installed from PyPI, so in that case you would have to use pip in your conda env, which is again slower than uv.
-
Docker Setup Tutorial for LLMs: Last Resort Option Due to Overhead
By
–
There's a tutorial for that as well: https://
github.com/rasbt/LLMs-fro
m-scratch/tree/main/setup/03_optional-docker-environment
…
But I would only consider that as the last resort because it adds overhead and is not very resource efficient. -
Popular trend emerging in machine learning circles
By
–
it might take years, but I think it's already pretty popular (in my ML circles at least).
-
TorchTune Library Growth Inspired by LitGPT
By
–
Oh yeah, torchtune is a good library. It was modeled after our LitGPT library and has grown quite a bit!
-
DevContainers Setup Guide and Local Machine Execution Preferences
By
–
There's a setup guide for DevContainers here, too: https://
github.com/rasbt/LLMs-fro
m-scratch/tree/main/setup/03_optional-docker-environment
…
No offense, but personally I had running it on my local machine. It has slower startup times, adds overhead, and is also storage-inefficient. -
Comparing Energy Consumption: Video Streaming vs AI Model Inference
By
–
Watching 20 min of a 4k video on YT is ~0.5 kWh Running 1000 Llama 3 8B prompts on a MacBook Pro: ~0.0025 kWh Running 1000 GPT 4o prompts: ?
-
Separate QKV Matrices Offer Better Big O Performance
By
–
Yes that's fair, in big O terms the separate QKV should be faster
-
GPU Precision Evolution: From 32-bit to FP8 in Transformers
By
–
Yes. Originally GPUs were designed for 32 bit precision, but in transformer contexts, you get away with even lower precision (16 bit is the standard usually). In recent months, people pushed that even further to FP8. One of the most recent architectures that successfully did FP8