Given that you are using nn.Linear, also looking at nanoGPT . (Although here, I guess the reimplementation serves educational purposes)
@rasbt
-
Why Do Developers Avoid PyTorch’s Built-in Transformer Layers?
By
–
Yeah, I always wonder what's up with that. The various Transformer layers in PyTorch have been around since ~2020, but everyone seems to be (still) rolling their own, https://
pytorch.org/docs/stable/nn
.html#transformer-layers
… Are the implementations somehow bad or inefficient? Genuine question. -
Jensen-Shannon Divergence: DNN Training Reproducibility Issues on Nvidia GPUs
By
–
Haha good one . Jensen-Shannon divergence: Training DNNs on Nvidia GPUs is not reproducible due to randomly selected code
-
Question about TPU and Nvidia GPU compatibility for suggestion
By
–
Thanks for the suggestion! That's only for TPUs, or would it also work on Nvidia GPUs?
-
Considering accuracy trade-offs and LoRA alternatives
By
–
Thanks for suggesting! Yeah, I was thinking about that as well, but it may affect accuracy, and then once going this route, one might as well also think about LoRA etc.
-
Helpful PR from Adrian Waelchli on Lightning Repository
By
–
You might find @adrianwaelchli 's PR helpful: https://
github.com/Lightning-AI/l
ightning/pull/17462
… -
Cost-Effective Hardware Selection Techniques Reduce Cloud Server Expenses
By
–
I'd say these techniques will allow you to choose cheaper hardware, which will essentially reduce your cloud server costs.
-
Fabric’s Intuitive API for PyTorch Script Modifications
By
–
I can't say much about accelerate since I never really used it, but regarding Fabric, I like the pretty intuitive API. It's basically just a minimal number of lines of code to modify a PyTorch script.
-
Vision Transformers: Tokenizing Images Like Language Models
By
–
In a tweet: similar to LLMs (language transformers) but tokenizing images instead of text:
-
Modern Classification: Multiple Output Nodes vs Sigmoid Threshold
By
–
So, in this case, rather than using a threshold on a logistic sigmoid score [0, 1], in most modern architectures, you usually define at least 2 output nodes. Then, you determine the class based on which output node has the highest score.