Bay Area Friends: Join us tomorrow at Orchestrating #GenAI Apps Meetup with NVIDIA and NetApp. We'll be discussing what it takes to build an #LLM serving platform at scale. Best of all it's free to attend, save your spot:
LLMS
-
Why 10B Tokens Suffice for GPT Training Performance
By
–
Great question yes I was surprised that 10B seemed enough. I believe GPT-2 was trained on somewhere ~100B tokens. The reason we reach this performance in 10B tokens I think may be the following: 1. FineWeb could just be higher quality than WebText, on a per-token basis. This was
-
GPT-3 Hyperparameters Analysis and Model Scaling Expectations
By
–
ah ok these are gpt3 hparams. it sounds like this alone would be enough to beat GPT2, and its unknown if 290B more tokens would get it to match or beat GPT-3 Small looking forward to the 1.5b – fascinating to see these all documented and taught live!!
-
FineWeb Dataset: Improvements Over GPT-2 Training Data
By
–
10B tokens of FineWeb! Ilya said WebText was 40B tokens (
https://
youtube.com/watch?v=13CZPW
mke6A&t=3645s
… – for gpt2 1.5b) what accounts for the improved loss/accuracy that you got over GPT2 – have we improved our dataset filtering? were there smarter hparam choices made here? any ballpark attributions -
Start Latency Issues with Dataset Tokenization in Python
By
–
TIL, will look into! The thing that makes this a bit complicated right now is the start latency. What bloats up the setup time right now is the dataset and its tokenization, which is all done in Python right now. Installing huggingface datasets, downloading FineWeb 10B and
-
Anthropic vs OpenAI: The AI Competition Behind ChatGPT Launch
By
–
Para quien no ubique a Jan, aquí el salseo de la semana pasada. Recordemos también que Anthropic se fundó de equipos disidentes de OpenAI. Y recordemos también que OpenAI aceleró la salida de ChatGPT por miedo a Anthropic adelantándose con algo similar.
-
Training Loss Noise and Validation Loss Smoothing Explained
By
–
Training loss is evaluated over the batch, i.e. 0.5M tokens. It's noisy but this is expected, you could be iterating through easy or hard documents in the training data. The validation loss is averaged over 20 batches of 0.5M tokens (this is a hyperparameter), so it is smoother.
-
Conservative hyperparameters with potential for 10X speedup
By
–
Agree!! I'm using very conservative settings for a lot of the hyperparameters (following GPT-3 paper when possible) and haven't tried to speed this up at all yet, but I expect a 10X multiplier here should be possible.
-
Explore and Compare All LLMs with ChatLLM Teams
By
–
Play around and compare and contrast all LLMs with ChatLLM Teams!https://t.co/5QdaueRsfN pic.twitter.com/avSTiUwLDQ
— Abacus.AI (@abacusai) 28 mai 2024Play around and compare and contrast all LLMs with ChatLLM Teams! https://
chatllm.abacus.ai -
Reproduce GPT-2 124M in llm.c for $20 in 90 Minutes
By
–
# Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 The GPT-2 (124M) is the smallest model in the GPT-2 series released by OpenAI in 2019, and is actually quite accessible today, even for the GPU poor. For example, with llm.c you can now reproduce this model on one 8X