Quick one – when you say “Nearly every company in the Sequoia network is building language models into their products.” Do you refer to the entire Sequoia portfolio or the 33 companies you selected to sample, which I assume are those most likely to be in a spot where LLMs are
LLMS
-
Efficient Falcon Finetuning with LoRA on Single GPU
By
–
Have been heads-down porting LoRA (low-rank-adaptation) to finetune Falcon more efficiently and ran some performance benchmarks. My longer write-up here: https://
lightning.ai/pages/communit
y/finetuning-falcon-efficiently/
… Long story short: you can finetune Falcon in 1 h on a 52k dataset using a single GPU with 16 GB RAM. -
LLaMA-Adapter Compatible with Falcon Models
By
–
(And yes, that's not a typo, you can use LLaMA-Adapter for Falcon :P)
-
LoRA vs LLaMA-Adapter: Runtime Performance Comparison
By
–
I'd suggest LoRA or LLaMA-Adapter. Either one works fine and has almost exactly the same runtime. Will upload a benchmark later today.
-
Model Contributions Comparison: Historical Performance Analysis
By
–
They contributed far more a while ago than gpt4
-
Distribution and Compute as AI Competitive Moats
By
–
It's pretty obvious to me that whoever has the distribution and compute wins. Distribution allows you to acquire customers at zero and compute allows you to discount beyond other providers. Model/data is not a moat. If you build something good people will train on your outputs.
-
Parameter-Efficient Fine-Tuning of Pretrained Large Language Models
By
–
Now, if we talk about parameter-efficient finetuning of a pretrained LLM, that's a different story.
-
LLMs limitation acknowledged in discussion
By
–
Yeah, agreed. But unfortunately, that wouldn't be possible with LLMs.
-
Exploring 100K Context Windows in AI Applications
By
–
We're excited to see how people are using 100K context windows! https://t.co/cnh4EX5wkd
— Anthropic (@AnthropicAI) 14 juin 2023We're excited to see how people are using 100K context windows!
-
RLHF Training Method Reduces Human Rater Involvement Requirements
By
–
Yes and afaik the way they use RLHF requires less involvement from human raters, will try to find something