Since the model can't really explicitly distinguish between synthetic and non-synthetic data during training, the best way would tackle the problem at the root: ensuring that the synthetic data-generating model does not produce hallucinated contents.
@rasbt
-
Microsoft’s Phi-4 14B Model Weights Now Publicly Available in LitGPT
By
–
The model weights of Microsoft’s phi-4 14B LLM are now finally publicly available. Thanks to a kind community contribution, it’s now also available in LitGPT already (litgpt finetune microsoft/phi-4):
https://
github.com/Lightning-AI/l
itgpt/pull/1904
… I will have more to say about phi-4 in my upcoming AI -
December 2024 AI Research Papers Added to List
By
–
In case you were looking for the December 2024 AI research papers, I just added them to the list! https://
magazine.sebastianraschka.com/i/152305086/de
cember
… -
LoRA Tutorial and Additional Tips from Readers
By
–
Glad you found it useful, and thanks for the kind words! I think you probably already read that, but there is a LoRA from-scratch tutorial in the last Appendix. There was a reader how also had some useful additional tips regarding LoRA:
-
No Need for Databases in LLM Pretraining Due to Embedding Changes
By
–
I don't think you need databases to pretrain LLMs because you can basically throw out the embedding database after each batch since the embedding layer changes after each training iteration.
-
GPT-2 Foundation with Llama 3 Advanced Implementation Guide
By
–
The book is GPT-2 based, which is more beginner friendly. The Llama 3 code is bonus material. (It makes sense to implement GPT-2 first and than convert that to Llama 3 imho.) Llama 3 is dense, not an MoE though.
-
AI Research Highlights 2024: Mixture-of-Experts and LLM Scaling Laws
By
–
Happy New Year! To kick off the year, I've finally been able to format and upload the draft of my AI Research Highlights of 2024 article. It covers a variety of topics, from mixture-of-experts models to new LLM scaling laws for precision:
-
Paperback Book Sale Announcement on Amazon
By
–
Haha, thanks for the shout out!! Btw the paperback is currently on sale on Amazon in case that helps (
https://
amazon.com/Build-Large-La
nguage-Model-Scratch/dp/1633437167/ref=tmm_pap_swatch_0
…) -
V100 GPUs Originally Designed for Molecular Dynamics Simulations
By
–
I am old enough to remember that V100 were actually designed with MD simulations in mind (or maybe it was their predecessor?) due to their fp64 support
-
Use Cases for Higher Precision Computing: Debugging, Education, Simulations
By
–
Debugging, educational use cases, maybe simulations you run in higher precisions (like MD and physics simulations)