AI - AI Dynamics

→ View original post on X — @thegautamkamath

Grant funding allocation practices in research

By

@thegautamkamath

–

13 April 2023 20h38

Technically, we put this down on grant applications. But then they ignore it and give however much they planned to anyway.

Chain-of-Thought Prompting Enables Multi-Step Reasoning in Language Models

By

–

13 April 2023 20h32

3 (cont). One way to elicit reasoning is via "chain-of-thought (CoT) prompting", which gives examples of intermediate reasoning steps in-context. CoT prompting enables large LMs to do multi-step reasoning tasks, increasing the range of tasks that LMs can do.

Reasoning: The Key Differentiator Between Classical ML and Intelligence

By

–

13 April 2023 20h32

3. The last idea is reasoning, which differentiates classical ML techniques from intelligence. Classical ML approaches need a lot of data and are black-box. Intelligent agents learn from a few examples and can do abstract reasoning.

Untested Abilities and Emergent Phenomena in Scaling Large Language Models

By

–

13 April 2023 20h32

2C. Since we haven't tested all possible abilities, we don't know the full range of abilities that have emerged in large language models.
2D. We're likely to see more emergent phenomena as we continue to scale up models (and implicit argument for more scaling).

Unpredictable Emergence in Language Models: Key Implications

By

–

13 April 2023 20h32

There are at least four profound implications of emergence:
2A. Emergence cannot be predicted simply by extrapolating the scaling curves from smaller models.
2B. Emergent abilities are not explicitly specified by the trainer of the language model.

Emergence: Large Language Models Gaining Unexpected Complex Abilities

By

–

13 April 2023 20h32

2. Emergence is a phenomenon where large language models gain abilities that are not present in smaller language models. An example of an emergent ability is doing complex math questions.

Scaling Laws: Model Size, Data, and Compute for LM Improvement

By

–

13 April 2023 20h32

Key takeaways: 1. Scaling involves increasing model size, data, and compute. Scaling is challenging (cost, infra, etc), but important, since "scaling laws" tell us that scaling predictably makes LMs better.

Three Ideas Driving the LLM Revolution: Scaling, Emergence, and Reasoning

By

–

13 April 2023 20h32

I gave an invited lecture at New York University for @hhexiy
's class! I covered three ideas driving the LLM revolution: scaling, emergence, and reasoning. I tried to frame them in a way that reveals why large LMs are special in the history of AI. Slides: