It's been a while and would have to check again, but wasn't BigBird a encoder LLM? Versus this is a decoder-style LLM for generating texts. But generally yes!
@rasbt
-
LongNet: Scaling Transformers to 1 Billion Tokens with Dilated Attention
By
–
We had 1) the RMT paper on scaling Transformers to 1M tokens, and 2) the convolutional Hyena LLM for 1M tokens. How about 3) LONGNET: Scaling Transformers to 1 Billion Tokens (
https://
arxiv.org/abs/2307.02486)!? It achieves linear (vs quadratic) scaling via dilated (vs self) attention. -
Open Source Algorithm vs Trade Secrets Claims
By
–
Not sure what they include in "trade secrets", but last time I checked "the algorithm" was open source
-
Memory-Lowering Techniques for PyTorch Hardware Optimization
By
–
Depending on the hardware, and next to parameter-efficient finetuning, there are a bunch of memory-lowering techniques you can try. Summarized them here: https://
github.com/rasbt/pytorch-
memory-optim
… -
LLM Guidelines for Training, Finetuning, and Inference
By
–
I am not aware of a guideline, but it usually also depends on what you are after: (pre)training, finetuning, inference. We have a unified codebase for several LLMs you can try here: https://
github.com/Lightning-AI/l
it-gpt
… -
ViT-everything: Summarizing the main theme
By
–
Thanks! Hah, and to maybe summarize the main theme so far: ViT-everything!
-
CVPR 2023 Main Themes: Vision Transformers, Diffusion Models, NeRFs
By
–
Felt the urge to write about computer vision for a change! Summarizing the main themes from CVPR:
1) Vision transformers
2) Diffusion models
3) NeRFs
4) Object detection & segmentation Recapping it in more detail along with some paper highlights here: -
Recommendation for RLHF Technical Tutorial
By
–
Great collection! And after the technical tutorials, I can highly recommend “How RLHF actually works”
-
What’s wrong with miniconda/miniforge?
By
–
What's wrong with miniconda/miniforge? https://
github.com/rasbt/scipy202
3-deeplearning/tree/main/00-1_python-setup-guide
… -
Open-sourcing LLaMa weights constrained by dataset legal concerns
By
–
Yup, totally agree. That’s probably it. Afaik the reason why they couldn’t open-source the LLaMa weights was the dataset. So instead of scraping data from elsewhere and staying on shaky legal grounds, getting data from your own platform is probably easiest.