Interesting! I'd say the nn.TransformerEncoderLayer and nn.TransformerDecoderLayer classes etc. actually precede most tutorials out there. Chicken-egg problem maybe
CODE
-
Using nn.Linear and nanoGPT for Educational Implementation
By
–
Given that you are using nn.Linear, also looking at nanoGPT . (Although here, I guess the reimplementation serves educational purposes)
-
Why Do Developers Avoid PyTorch’s Built-in Transformer Layers?
By
–
Yeah, I always wonder what's up with that. The various Transformer layers in PyTorch have been around since ~2020, but everyone seems to be (still) rolling their own, https://
pytorch.org/docs/stable/nn
.html#transformer-layers
… Are the implementations somehow bad or inefficient? Genuine question. -
Jensen-Shannon Divergence: DNN Training Reproducibility Issues on Nvidia GPUs
By
–
Haha good one . Jensen-Shannon divergence: Training DNNs on Nvidia GPUs is not reproducible due to randomly selected code
-
GitHub Guide: Tips for Working with Large Language Models
By
–
GitHub – brexhq/prompt-engineering: Tips and tricks for working with Large Language Models like OpenAI's GPT-4. https://bit.ly/46l3eoi #AI #MachineLearning #DeepLearning #LLMs #DataScience
-
Considering accuracy trade-offs and LoRA alternatives
By
–
Thanks for suggesting! Yeah, I was thinking about that as well, but it may affect accuracy, and then once going this route, one might as well also think about LoRA etc.
-
Helpful PR from Adrian Waelchli on Lightning Repository
By
–
You might find @adrianwaelchli 's PR helpful: https://
github.com/Lightning-AI/l
ightning/pull/17462
… -
Fabric’s Intuitive API for PyTorch Script Modifications
By
–
I can't say much about accelerate since I never really used it, but regarding Fabric, I like the pretty intuitive API. It's basically just a minimal number of lines of code to modify a PyTorch script.
-
Python AI Feature Successfully Ported to JavaScript
By
–
Was awesome to see this in python, glad it made its way to js! https://t.co/qGwE4ChYgC
— Harrison Chase (@hwchase17) 3 juillet 2023Was awesome to see this in python, glad it made its way to js!
-
Modern Classification: Multiple Output Nodes vs Sigmoid Threshold
By
–
So, in this case, rather than using a threshold on a logistic sigmoid score [0, 1], in most modern architectures, you usually define at least 2 output nodes. Then, you determine the class based on which output node has the highest score.