AI - AI Dynamics

having fun optimizing minGPT today
– base: 495ms
– zero_grad(set_to_none=True): 492
– torch.jit.script gelu: 463
– OMP_PROC_BIND=CLOSE: 453
– torch.backends.cuda.matmul.allow_tf32: 143
– torch.autocast(torch.bfloat16): 121
– FlashAttention: 102
now: more fused kernels more better

→ View original post on X — @karpathy

27 December 2022

Splitting minGPT into educational and efficient versions

By

@karpathy

–

27 December 2022 18h32

Context I realized I have to split up minGPT because I can't properly simultaneously satisfy both 1) educational and 2) efficient in one repo. So I'm separately writing 1) the maximally educational minGPT (+video etc.) and 2) a more efficient (still ~clean) version that has teeth

→ View original post on X — @karpathy

27 December 2022

Thank You for Supporting AI Research and Innovation in 2022

By

@aiatmeta

–

27 December 2022 18h31

From all of us to the entire community: Thank you for supporting AI research, innovation & open science in 2022 — we're excited to continue pushing this (and more) forward together in 2023.

→ View original post on X — @aiatmeta

27 December 2022

Dealing with GPT-3 Limited Context Window Constraints

By

@swyx

–

27 December 2022 18h04

how do you deal with the limited context window of GPT3? afaik, ChatGPT a limit of 8092 tokens, whereas GPT3 is lower (3-4k?) any langchain solution to that sort of thing?

→ View original post on X — @swyx

27 December 2022

Top 50 Public Datasets for Machine Learning

By

@mit_csail

–

27 December 2022 18h00

The 50 best public datasets for machine learning: http://
bit.ly/50datasets (v/
@dd_invest
)

→ View original post on X — @mit_csail

27 December 2022

Language Models Achieve State of the Art on Medical QA

By

@_jasonwei

–

27 December 2022 17h09

Language models achieve state of the art on medical QA, demonstrating the power of scaling + prompting. Excited to see the emergent use of language models in domain-specific applications in the next few years!

→ View original post on X — @_jasonwei

27 December 2022