The norm/wd paper showed iirc that you can get exactly the same effect with an lr scheduler. I wonder if we'd get better results by carefully tuning our schedulers instead of using wd? It's something I've wondered about for years but never got around to…
@jeremyphoward
-
Weight Decay Ineffectiveness in Llama and Mistral Models
By
–
Note that Llama and Mistral have parameterised norm layers, so weight decay has no regularizing effect
-
Creative experimentation with DALL-E 3 image generation
By
–
My wife wanted me to show her Dall-e 3, and for some reason this was the first thing that jumped into my head
-
Open-Orca Fine-Tune Model Inference Implementation
By
–
It's an open-orca fine-tune https://
kaggle.com/code/ybabakhin
/1st-place-single-model-inference
… -
Winning Kaggle Science QA Competition with 7B Model
By
–
In the recent Kaggle science QA competition the winning model was a 7B 😀
-
H100 GPU Pricing Comparison Lambda Labs Alternative
By
–
I might be missing something – afaict they're 5x more expensive than lambda labs for H100 for instance
-
GPU Performance Comparison Mojo Programming Language
By
–
You're using GPU though, aren't you? (Which is still cool – but not really a great comparison AFAICT; when Mojo adds GPU support that would be cool to compare.)
-
Executive Centralization Reduces Organizational Adaptability and Flexibility
By
–
There seems to be a strong temptation for execs to centralise and control things, leading to decreased adaptability and flexibility in their org.
-
Mergers Destroy Value Despite Synergy Optimization Efforts
By
–
There is some empirical support for this claim too — e.g on average, mergers destroy value (i.e the value of the combined company is lower than the sum of the original companies). This is despite mergers being explicitly chosen for maximising synergies.
-
Iterative Token-Thought Loop Architecture for LLMs
By
–
Hmm what if you did token->thought1->token->thought2->token->thought3->token->thought4->token->thought1->token->thought2… ?