For transformers text decoders it's not clear yet afaik – but at least having the encoder head at a higher lr seems to be reliable. I also suspect the first 2 and last 3 layers in the body should have higher lr but I haven't got rigorous tests.
RESEARCH
-
Video-to-video adversarial networks advance generative AI capabilities
By
–
video2video adversarial https://
x.com/StrangestMedia
/StrangestMedia/status/1715436402215997851
… -
SAS Explore On-Demand: AI Analytics and Data Insights
By
–
See all the moments that you wish you hadn't missed from SAS Explore, on the topics you love: artificial intelligence, analytics, data and industry insights . All-access to our top #ExploreSAS content is here on-demand for a limited time http://
2.sas.com/6011uwK3L #DataAnalytics -
Learning Rate Schedules and Discriminative Training in Deep Learning
By
–
Yeah this was one of my favourite papers of the year. It doesn't however investigate the LR schedule angle – which would require discriminative LR to be used (which basically no one bothers with nowadays for some reason)
-
Weight Decay Regularization and Parameterised Norm Layers
By
–
This was the paper that pointed out wd doesn't regularise in the presence of parameterised norm layers, and effectively just adjusts the LR https://
arxiv.org/abs/1706.05350 -
Weight Decay vs Learning Rate Scheduling in Model Training
By
–
The norm/wd paper showed iirc that you can get exactly the same effect with an lr scheduler. I wonder if we'd get better results by carefully tuning our schedulers instead of using wd? It's something I've wondered about for years but never got around to…
-
The Boundless Demand for AI-Powered Insights
By
–
The boundless demand for insight | Hex https://
bit.ly/3Poo1Qb #AI #MachineLearning #DeepLearning #LLMs #DataScience -
DALL-E 3 Paper: OpenAI Improves Image Generation Captions
By
–
DALL·E 3 Explained: Improving Image Generation with Better Captions This week's video focuses on huge news: the release of the DALLE 3 paper by OpenAI! I dissected it and gave you all the cool insights to get from this super interesting research paper! Spoiler: they tell us
-
Weight Decay Ineffectiveness in Llama and Mistral Models
By
–
Note that Llama and Mistral have parameterised norm layers, so weight decay has no regularizing effect
-
Critique of Computer Vision Field’s Military and Surveillance Applications
By
–
I finally have this book, written by James E. Dobson, professor of creative writing & English. Good thing we're getting rid of the humanities am I right? Explains my aversion to the computer vision field in which I got my PhD. All military & surveillance.