"A self-attention model is a linear MLP with dynamic weights" sounds about right
@rasbt
-
Open Source LLMs Superior to GPT-4 API for Critical Services
By
–
I am just picturing a case where someone built a crucial service for their company on top of the GPT-4 API . This is a good example why open source LLMs > GPT-4.
-
GPT-4 Degradation and Alternative Model Architectures
By
–
Oh, they made GPT-4 worse … again? Dang. I finally started to find it useful.
Thanks for the reference to Retention, looks like it's orthogonal attempt to RWKV and Hyena? -
What Major AI Developments Happened Last Week?
By
–
Just got back from a week in nature, and wow, it seems like everyone and everything went full steam ahead:
– LLaMA 2
– FlashAttention 2
– …
What else did I miss last week? -
Writing Technical Books for LLMs Instead of Humans
By
–
I would still write books. I guess the difference is that I’d write them for LLMs rather than humans, who probably don’t need technical books anymore then
-
Serial-Position Effect in Transformer Training Data
By
–
Yup! > The serial-position effect plays a role in understanding how humans develop short- and long-term memory. My guess is the transformer picks this up from the training data, which reflects how humans write articles as well.
-
kNN and Gzip paper coverage planned after vacation
By
–
*PS: I didn't include the kNN + Gzip paper in this one since the paper was originally uploaded to arxiv in December 2022 https://
x.com/rasbt/status/1
679680423671001090?s=20
… But I have another "special coverage" for that one in mind in a few weeks once I am back from vacation 🙂 -
LLM Context Lengths and Efficient Transformers: June-July Research Highlights
By
–
From long context lengths in LLMs to more efficient transformers for diffusion models. It's been another month with lots of interesting research. I put together an annotated list of 24 highlights from June to July:
-
PyTorch Python Overhead Estimated at Approximately 10 Percent
By
–
I think the Python overhead in PyTorch is just ~10% as per PyTorch benchmarks compared with the non-Python API.
-
GPU Support Request for PyTorch Integration
By
–
I hope it will support GPUs some time so we can use it with PyTorch etc. (Thanks for the link, but sry, as a personal thing, I am not watching content from that interviewer due to his blocking spree of AI and deep learning researchers and content creators)