*PS: I didn't include the kNN + Gzip paper in this one since the paper was originally uploaded to arxiv in December 2022 https://
x.com/rasbt/status/1
679680423671001090?s=20
… But I have another "special coverage" for that one in mind in a few weeks once I am back from vacation 🙂
@rasbt
-
kNN and Gzip paper coverage planned after vacation
By
–
-
LLM Context Lengths and Efficient Transformers: June-July Research Highlights
By
–
From long context lengths in LLMs to more efficient transformers for diffusion models. It's been another month with lots of interesting research. I put together an annotated list of 24 highlights from June to July:
-
PyTorch Python Overhead Estimated at Approximately 10 Percent
By
–
I think the Python overhead in PyTorch is just ~10% as per PyTorch benchmarks compared with the non-Python API.
-
GPU Support Request for PyTorch Integration
By
–
I hope it will support GPUs some time so we can use it with PyTorch etc. (Thanks for the link, but sry, as a personal thing, I am not watching content from that interviewer due to his blocking spree of AI and deep learning researchers and content creators)
-
Mojo programming language benefits for Rust developers
By
–
If someone likes programming in Rust, how would Mojo help here?
-
Computing Fragmentation Creates Barriers for Deep Learning Education
By
–
Sure, but a lot of people don’t have that kind of time. If the computing ecosystem would be that fragmented, the prerequisites for a intro to deep learning class would be insane
-
Transformers Are Popular But Simple Baselines Still Matter
By
–
Yeah. I doubt that there will be a follow-up paper though. I think the point was more like "transformers are the shiny new toy, but don't forget about also implementing simple baselines when you start a new project".
-
Disclosure Differences Between LLMs and BERT Models
By
–
Interesting point. For LLMs like GPT-3 & 4 we simply don't know because it's not disclosed. For BERT we do know though. There's separate BERT-English and BERT-multilingual models, for example, etc.
-
Gzip and kNN: Text Similarity Through Compression
By
–
In short, the idea behind this gzip + kNN approach is that a given text A is close to the size of two concatenated texts A + B when compressed using Gzip. Or, the compressed size of A+A is similar to the compressed size of just A (plus a constant value for the back reference)
-
Sequence Packing Applied to Image Patches – Novel Approach
By
–
So, basically sequence packing but for image patches. Neat. And surprised no one has tried that yet