The main options are fp32 (or tf32), bf16, or fp16+mixed prec afaik.
@jeremyphoward
-
FP16 Training Optimization: Dynamic Loss Scaling Without Mixed Precision
By
–
I suspect the problem you're testing here might be too easy to see differences – you're getting nearly 100% either way. Would be interesting to see fp16 with dynamic loss scaling but without mixed precision to compare to there.
-
BF16 vs FP16 Training Performance Comparison Analysis
By
–
There was some comparison of training performance by @StasBekman recently that showed bf16 non-mixed training was quite a bit worse. Which I'd expect, because bf16 has much less precision than even fp16.
-
Generative Models and Dangerous Compound Synthesis Safety Concerns
By
–
I don't get it, personally. They "chose to drive the generative model towards compounds such as the nerve agent VX" — and the model did that pretty well. E.g it recreated VX. VX is bad – but there's already loads of public info on how to make it.
-
BF16 Benefits Beyond Loss Scaling in Mixed Precision
By
–
Other than avoiding loss scaling, are there other benefits to using bf16? It still seems to require mixed precision afaict, right?
-
Fine-tuning LLMs: Local Communities and Development Resources
By
–
I don't find much great info on *doing* fine-tuning at r/localllama, but I find the info there on *using* fine-tunes really great. The folks that create the tunes that r/localllama do their development in the open on various discords (e.g. alignment labs, nous, skunkworks)
-
Using Standard Vision Model Encoders for Your Projects
By
–
For the encoder just use whatever works for your normal vision models I think.
-
Transformer Learning Rates: Encoder Head and Layer Optimization
By
–
For transformers text decoders it's not clear yet afaik – but at least having the encoder head at a higher lr seems to be reliable. I also suspect the first 2 and last 3 layers in the body should have higher lr but I haven't got rigorous tests.
-
Learning Rate Schedules and Discriminative Training in Deep Learning
By
–
Yeah this was one of my favourite papers of the year. It doesn't however investigate the LR schedule angle – which would require discriminative LR to be used (which basically no one bothers with nowadays for some reason)
-
Weight Decay Regularization and Parameterised Norm Layers
By
–
This was the paper that pointed out wd doesn't regularise in the presence of parameterised norm layers, and effectively just adjusts the LR https://
arxiv.org/abs/1706.05350