MACHINE LEARNING

→ View original post on X — @lilianweng

Transformer Inference Optimization: Reducing Computational Costs

By

@lilianweng

–

11 January 2023 23h43

Large Transformers are powerful but expensive to train & use. The extremely high inference cost is a big bottleneck for adopting them for solving real-world tasks at scale. Check out my new post on some ideas on inference optimization for Transformers:

→ View original post on X — @aibreakfast

GPT-3.5 passes multiple-choice sections of Bar Exam for Evidence and Torts

By

@aibreakfast

–

11 January 2023 23h35

GPT-3.5 took the Bar Exam "Passing range" scores on multiple-choice section for Evidence & Torts categories. Full write-up in Monday's newsletter Research paper: https://
arxiv.org/pdf/2212.14402
.pdf
…

→ View original post on X — @ingliguori

Industry 4.0: The Evolution of Maintenance Strategy

By

@ingliguori

–

11 January 2023 21h20

Industry 4.0 – The evolution of Maintenance Strategy
Check out https://
bit.ly/3GO1IQz
By @ingliguori #DigitalTransformation #Cloud #BigData #Industry40 #cybersecurity #Blockchain #DX #Analytics #AI #IIoT #DataScience #IoT #IoTPL #SmartManufactory #RPA #Robotics #MLOps

Comparing Modern GPT Models to OpenAI’s Early GPT-2 Capabilities

By

–

11 January 2023 21h15

It’s about as good as OpenAI’s baby GPT-2 from ~4 years ago. (Their paper at that time had models from 124M to 1.3B). Today’s bleeding edge GPTs reach scale (in model size and data size) that requires significant infrastructure and further finetuning to align them (RLHF etc).

Clarification on GPT-2 Model Size: 124M vs 1.3B Parameters

By

–

11 January 2023 20h47

Careful this is the 124M model. The biggest GPT-2 was 1.3B

Neural Networks: Zero to Hero Series – Building Networks from Scratch

By

–

11 January 2023 20h19

(This will be part of my ongoing series Neural Networks: Zero to Hero https://
karpathy.ai/zero-to-hero.h
tml
… , on building neural networks, from scratch, in code. I have tweeted some of these videos individually already)

Future Plans for GPT-2 Implementation and Educational Content

By

–

11 January 2023 20h04

I'd like to continue to make it faster, reproduce the other GPT-2 models, then scale up pre-training to bigger models/datasets, then improve the docs for finetuning (the practical use case). Also working on video lecture where I will build it from scratch, hoping out in ~2 weeks.