MACHINE LEARNING

→ View original post on X — @miketamir

Your Guide to AI: June 2023

By

@miketamir

–

04 July 2023 3h01

Your guide to AI: June 2023 https://
bit.ly/3JfgReE #AI #MachineLearning #DeepLearning #LLMs #DataScience

Why Transformers and Self-Attention Over Convolutional or RNN Layers

By

–

04 July 2023 1h19

Sure. But why specifically for transformer layers and self-attention, not say convolutional or RNN layers?

Multiquery Attention Challenges with LoRA Porting Between Models

By

–

04 July 2023 0h30

That’s absolutely true and a good point. I remember multiquery attention struggles when trying to port LoRA from LLaMA to Falcon.

TransformerEncoderLayer Abstraction Compared to Conv2D and RNN

By

–

03 July 2023 22h23

Ok, fair. But if you look at the TransformerEncoderLayer class, for example, I don't think it's more abstract than nn.Conv2D or nn.RNN, for example. For some reason people just like implementing Transformer layers from scratch way more than convolutional or recurrent layers .

Everyone’s Using nn.Linear and Similar Neural Network Layers

By

–

03 July 2023 22h14

But everyone's using nn.Linear etc.

PyTorch Transformer Classes Predate Most Online Tutorials

By

–

03 July 2023 22h01

Interesting! I'd say the nn.TransformerEncoderLayer and nn.TransformerDecoderLayer classes etc. actually precede most tutorials out there. Chicken-egg problem maybe

→ View original post on X — @replit

Quadratic AI Training Costs: How Engineers Build Bigger Models Affordably

By

@replit

–

03 July 2023 22h00

Did you know that the cost of training an AI model increases approximately quadratically as you increase the number of tokens and the model size? ML engineers are still building bigger models without breaking the bank. pic.twitter.com/tB7EiintQl
— Replit ⠕ (@Replit) 3 juillet 2023

Did you know that the cost of training an AI model increases approximately quadratically as you increase the number of tokens and the model size? ML engineers are still building bigger models without breaking the bank.

Using nn.Linear and nanoGPT for Educational Implementation

By

–

03 July 2023 21h54

Given that you are using nn.Linear, also looking at nanoGPT . (Although here, I guess the reimplementation serves educational purposes)

Why Do Developers Avoid PyTorch’s Built-in Transformer Layers?

By

–

03 July 2023 21h51

Yeah, I always wonder what's up with that. The various Transformer layers in PyTorch have been around since ~2020, but everyone seems to be (still) rolling their own, https://
pytorch.org/docs/stable/nn
.html#transformer-layers
… Are the implementations somehow bad or inefficient? Genuine question.