That's actually a good point. But it would be quite trivial to restart the cosine schedule after each ~epoch or checkpoint
@rasbt
-
Multi-epoch training effectiveness for Vision Transformers
By
–
A counter-argument is that multi-epoch training works pretty well for ViTs.
-
Scaling LLM Performance: Token-Crisis Solutions and Diminishing Returns
By
–
They actually might be doing this already but yeah, probably diminishing returns.
Some insights in the "To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis" paper: https://
arxiv.org/abs/2305.13230 -
JAX Deep Learning API: Keras Backend and Framework Choices
By
–
The problem with JAX always on the deep learning API side. Imho, it needed something like Keras. Nice that it’s now a backend! Btw, do use pure JAX in Keras or one of the intermediate APIs (like Flax)?
-
RetentionNet and other model architecture examples mentioned
By
–
You are welcome! And yes, those two I mentioned above would be examples of that. Also RentionNet but that didn’t fit within the tweet limit
-
Transformers Can Be Applied to Tabular Non-Time Series Data
By
–
Yup there are transformers for tabular non-time series data
-
Transformers and LLMs: Beyond the Common Assumption
By
–
Not all transformers are LLMs, since transformers can also be used for computer vision. And not all LLMs are transformers as there are large language models based on – recurrent (RWKV, https://
arxiv.org/abs/2305.13048) – or convolutional (Hyena, https://
arxiv.org/abs/2302.10866) approaches. -
Bag-of-Words Results for IMDb Movie Review Dataset
By
–
Nice work! I also had the bag-of-words results for the IMDb movie review dataset in my post https://
magazine.sebastianraschka.com/p/large-langua
ge-models-and-nearest
… -
Open Source Licenses and Non-Commercial Restrictions Compatibility
By
–
Isn’t open source by definition incompatible with non-commercial restrictions. Ie if it has a non-commercial restriction it can’t be open source?
-
Training Personal Email Assistant with Conversation Variety
By
–
That’s a good point. Yeah if there is sufficient variety in the conversations, I can see how one can train a personal email assistant that way.