LLMS - AI Dynamics

Rough example, a decent GPT-2 (124M) pre-training reproduction would be 1 node of 8x A100 40GB for 32 hours, processing 8 GPU * 16 batch size * 1024 block size * 500K iters = ~65B tokens. I suspect this wall clock can still be improved ~2-3X+ without getting too exotic.

→ View original post on X — @karpathy

11 January 2023

Future Plans for GPT-2 Implementation and Educational Content

By

@karpathy

–

11 January 2023 20h04

I'd like to continue to make it faster, reproduce the other GPT-2 models, then scale up pre-training to bigger models/datasets, then improve the docs for finetuning (the practical use case). Also working on video lecture where I will build it from scratch, hoping out in ~2 weeks.

→ View original post on X — @karpathy

11 January 2023

nanoGPT: Simplest Repository for Training Medium-Sized GPTs

By

@karpathy

–

11 January 2023 20h04

Didn't tweet nanoGPT yet (quietly getting it to good shape) but it's trending on HN so here it is 🙂 : https://
github.com/karpathy/nanoG
PT
…
Aspires to be simplest, fastest repo for training/finetuning medium-sized GPTs. So far confirmed it reproduced GPT-2 (124M). 2 simple files of ~300 lines

→ View original post on X — @karpathy

11 January 2023

ChatGPT Politeness and Data Training Practices Disagreement

By

@thatroblennon

–

11 January 2023 19h51

Fact – Saying thank you to ChatGPT improves responses later in the current chat session you're in. Fact – They have implied they are using data from the current free preview to train it to better respond in the future. I disagree with your assessment.

→ View original post on X — @thatroblennon

11 January 2023

The Growing Power of Text Prompts for AI Systems

By

@thatroblennon

–

11 January 2023 18h36

## Prompts as Commodities The power of a block of text has also never been bigger. The right choice of words can result in amazing outputs for AIs.

→ View original post on X — @thatroblennon

11 January 2023

LLMs With Tools: Beyond Single-Word Generation

By

@oriolvinyalsml

–

11 January 2023 18h31

Many people mistakenly think that large language models that generate one word at a time is the end game.
Connecting LLMs with tools, e.g.: search engines, python interpreters, etc., is super exciting, and leverages tools' power, robustness & more! pic.twitter.com/39PmdYEgwv
— Oriol Vinyals (@OriolVinyalsML) 11 janvier 2023

Many people mistakenly think that large language models that generate one word at a time is the end game.
Connecting LLMs with tools, e.g.: search engines, python interpreters, etc., is super exciting, and leverages tools' power, robustness & more!

→ View original post on X — @oriolvinyalsml

11 January 2023