what sets great builders apart is not that they are smarter [lots of people are smart], or even that they’re better at building it’s that they ask the right questions and focus on the right problems how do you get good at this? practice genuine, deep curiosity
RESEARCH
-
Focusing on Improving Modeling Performance Over Computational Performance
By
–
*the focus here is on improving modeling performance, not computational performance.
-
Position Interpolation Extends RoPE Context Windows to 32K Tokens
By
–
Rotary positional embeddings (aka RoPE) have been a recent cornerstone of modern LLM implementations since it supports flexible sequence lengths. In this paper, researchers propose Position Interpolation to increase RoPE-based context window sizes to 32,768 tokens 2/3
-
Humble approach to long context lengths with minimal finetuning
By
–
This sounds a bit more humble than the recent 1M or 1B context lengths claims. But this requires only minimal (1000 steps) finetuning. And it allows long document summarization using LLaMA 7B and 65B models, for example. Link to the paper here: https://
arxiv.org/abs/2306.15595 3/3 -
Extending Transformer LLMs Context Window via Positional Interpolation
By
–
In the last couple of days, we talked a lot about extending the context window of transformer LLMs. Here's one more: "Extending Context Window of Large Language Models via Positional Interpolation" 1/3
-
Falcon-40b-instruct Tops OpenLLM Leaderboard Rankings
By
–
Falcon-40b-instruct is the top performing model on the OpenLLM leaderboard. Built by @TIIuae
, it's free to use for commercial and research use. https://
huggingface.co/spaces/Hugging
FaceH4/open_llm_leaderboard
… -
LLMs Learning to Recognize Knowledge Gaps and Delegate Tasks
By
–
Soon, LLMs will know when they don’t know. They’ll know when to say IDK, or instead ask another ai, or ask a human, or use a different tool, or different knowledge base. This will be a hugely transformative moment.
-
Data Science Interview Questions and Answers
By
–
GitHub – youssefHosni/Data-Science-Interview-Questions-Answers: Curated list of data science interview questions and answers
https://bit.ly/42FHnF3 #AI #MachineLearning #DeepLearning #LLMs #DataScience -
Zero-Sum Performance Trade-offs in Attention Mechanisms
By
–
Yeah but my guess is it’s a zero sum issue. Like if you fixing the performance in the middle, you will probably have to sacrifice performance elsewhere. Otherwise if you pay attention to everything equally, you’ll lose the advantage of attention in a way.
-
Bookmarking Key Papers on Learning and Vision-Language Models
By
–
@chelseabfinn on learning to learn with gradients. I bookmarked it and hope to read it sometime: https://
ai.stanford.edu/~cbfinn/_files
/dissertation.pdf
… Also @karpathy on connecting images and texts, which was ahead of time given current progress in visual language learning: https://
cs.stanford.edu/people/karpath
y/main.pdf
…