Totally missed that and that sounds awesome. Genuine question, how do you leverage unified memory in PyTorch though? Was discussing this with a user who was using Blackwell GH200 the other day here:
@rasbt
-
Loading Pretrained Weights and Selecting Foundation Models
By
–
And in addition, as you will see later in chapter 5, it's necessary to load pretrained weights.
Now there are many open-weight models available, but my reason for this one was also that it's a) the basis for everything that came after (other architectures can be covered in -
Book Update: Future Plans for RNN, LSTM, and GAN Content Revision
By
–
Thanks for your interest in this. I hope to find the time some day, but it won't be very soon I'm afraid. I agree with your suggestions regarding RNN, LSTM, GANs (would replace them with more LLM and diffusion model contents). Unfortunately, the book was already at the max page
-
Synthetic Data Quality Limitations Compared to Raw Data
By
–
yes, totally. But my guess is model-generated synthetic data will always be a bit inferior then raw data (assuming raw data doesn't contain too much false information).
Because in the best case synthetic data preserves the information & in the worst case it adds hallucinations -
Synthetic Data for Pretraining: Proof-of-Concept Study
By
–
True, but I think it's more of a proof-of-concept / study of synthetic data for pretraining rather than generating a production-ready model.
-
Intersection of Pretrained LLM and Tool-Use Technology
By
–
Yeah, that's basically at the intersection of pretrained LLM and tool-use.
-
Synthetic Data Structure Benefits and Knowledge Limitations
By
–
Yeah, I think the main advantage is from the more refined nature of the synthetic data when it comes to response structure etc. Synthetic data can't really contain knowledge that raw data doesn't already include because the raw data was used to come up with the synthetic data in
-
Synthetic Data Format Benefits Versus Raw Data Knowledge Quality
By
–
Yes. I think the main benefit is that it comes in a more structured or refined format compared to raw data. But the knowledge is the same as in the raw data (and may even be more hallucinated), considering that raw data was used to generate the synthetic data through a model.
-
Synthetic Data from High-Quality Models as Transfer Learning Jumpstart
By
–
Yeah, I see it as some flavor of transfer learning (i.e., not starting from raw data). Synthetic data generated by a high-quality model (such as GPT-4o, which has already undergone extensive refinement) may serve as a kind of jumpstart to the model you are trying to train.