@rasbt - AI Dynamics - Page 81 of 95

Multiquery Attention Challenges with LoRA Porting Between Models

By

–

04 July 2023 0h30

That’s absolutely true and a good point. I remember multiquery attention struggles when trying to port LoRA from LLaMA to Falcon.

→ View original post on X — @rasbt

4 July 2023

TransformerEncoderLayer Abstraction Compared to Conv2D and RNN

By

@rasbt

–

03 July 2023 22h23

Ok, fair. But if you look at the TransformerEncoderLayer class, for example, I don't think it's more abstract than nn.Conv2D or nn.RNN, for example. For some reason people just like implementing Transformer layers from scratch way more than convolutional or recurrent layers .

→ View original post on X — @rasbt

3 July 2023

Everyone’s Using nn.Linear and Similar Neural Network Layers

By

@rasbt

–

03 July 2023 22h14

But everyone's using nn.Linear etc.

→ View original post on X — @rasbt

3 July 2023

PyTorch Transformer Classes Predate Most Online Tutorials

By

@rasbt

–

03 July 2023 22h01

Interesting! I'd say the nn.TransformerEncoderLayer and nn.TransformerDecoderLayer classes etc. actually precede most tutorials out there. Chicken-egg problem maybe

→ View original post on X — @rasbt

3 July 2023

Using nn.Linear and nanoGPT for Educational Implementation

By

@rasbt

–

03 July 2023 21h54

Given that you are using nn.Linear, also looking at nanoGPT . (Although here, I guess the reimplementation serves educational purposes)

→ View original post on X — @rasbt

3 July 2023

Why Do Developers Avoid PyTorch’s Built-in Transformer Layers?

By

@rasbt

–

03 July 2023 21h51

Yeah, I always wonder what's up with that. The various Transformer layers in PyTorch have been around since ~2020, but everyone seems to be (still) rolling their own, https://
pytorch.org/docs/stable/nn
.html#transformer-layers
… Are the implementations somehow bad or inefficient? Genuine question.

→ View original post on X — @rasbt

3 July 2023

Jensen-Shannon Divergence: DNN Training Reproducibility Issues on Nvidia GPUs

By

@rasbt

–

03 July 2023 21h11

Haha good one . Jensen-Shannon divergence: Training DNNs on Nvidia GPUs is not reproducible due to randomly selected code

→ View original post on X — @rasbt

3 July 2023

Question about TPU and Nvidia GPU compatibility for suggestion

By

@rasbt

–

03 July 2023 19h18

Thanks for the suggestion! That's only for TPUs, or would it also work on Nvidia GPUs?

→ View original post on X — @rasbt

3 July 2023

Considering accuracy trade-offs and LoRA alternatives

By

@rasbt

–

03 July 2023 19h04

Thanks for suggesting! Yeah, I was thinking about that as well, but it may affect accuracy, and then once going this route, one might as well also think about LoRA etc.

→ View original post on X — @rasbt

3 July 2023