MACHINE LEARNING

→ View original post on X — @swyx

Beyond Copilot: The Next Evolution in AI Developer Experience

By

@swyx

–

27 December 2022 22h10

The Big Question in AI-driven developer experience: What's after Copilot? Everyone is convinced this is "just" the beginning but can't articulate what the next killer form factor will be. Increasingly convinced next step change won't be from optimal stuffing context windows

Mixed Precision Training: Complexity Beyond High Level Concepts

By

–

27 December 2022 19h03

the high level picture is easy enough but keeping track of the mixed precision around the whole network, the dynamical behavior of the values and ranges, the support for them and their conversions across all the various kernels and library versions everywhere, is the nightmare

FP16 Efficiency and Gradient Scaling Considerations

By

–

27 December 2022 18h57

yeah fp16 is a little more efficient atm for the code as I have it right now but then need gradient scaler ;s

→ View original post on X — @swyx

Searching for canonical references to the bitter lesson

By

@swyx

–

27 December 2022 18h46

was looking for more references to the "bitter lesson" – is this the canonical source? http://
incompleteideas.net/IncIdeas/Bitte
rLesson.html
…

Splitting minGPT into educational and efficient versions

By

–

27 December 2022 18h32

Context I realized I have to split up minGPT because I can't properly simultaneously satisfy both 1) educational and 2) efficient in one repo. So I'm separately writing 1) the maximally educational minGPT (+video etc.) and 2) a more efficient (still ~clean) version that has teeth

Optimizing minGPT: Performance improvements from 495ms to 102ms

By

–

27 December 2022 18h32

having fun optimizing minGPT today
– base: 495ms
– zero_grad(set_to_none=True): 492
– torch.jit.script gelu: 463
– OMP_PROC_BIND=CLOSE: 453
– torch.backends.cuda.matmul.allow_tf32: 143
– torch.autocast(torch.bfloat16): 121
– FlashAttention: 102
now: more fused kernels more better

→ View original post on X — @mit_csail

Top 50 Public Datasets for Machine Learning

By

@mit_csail

–

27 December 2022 18h00

The 50 best public datasets for machine learning: http://
bit.ly/50datasets (v/
@dd_invest
)

→ View original post on X — @_jasonwei

Language Models Achieve State of the Art on Medical QA

By

@_jasonwei

–

27 December 2022 17h09

Language models achieve state of the art on medical QA, demonstrating the power of scaling + prompting. Excited to see the emergent use of language models in domain-specific applications in the next few years!