OpenAI triggered a Cambrian explosion in AI development with the release of ChatGPT. That was six months ago, and already developer efforts (especially in OSS) have shifted to focus on squeezing more performance out of smaller models. And even that's getting old!
@mattlynley
-
AI Regulation Challenge: Can Regulators Keep Pace With Breakthroughs
By
–
Sam Altman wants AI regulated, and indeed there will be regulation coming. But the technology is evolving so fast and breakthroughs are happening overnight that it's unclear if regulators—or OpenAI/Google, for that matter— can keep up.
-
Together Compute Raises $20M to Overcome Transformer Limitations
By
–
I spoke with @togethercompute which is exploring some new techniques to get past the Transformer wall. Together raised a $20m seed from Lux Capital and is also responsible for the RedPajamas open source dataset replicating LLaMA’s set.
-
MegaByte: Meta’s Latest Context Window Expansion Breakthrough
By
–
Finally a third one that is super recent is MegaByte, a paper coming out of Meta last week. This one has been getting a lot of buzz. The three approaches are attempts to expand the context window, which allows for more information coming in (and broader hidden prompts).
-
Sub-Quadratic Attention Techniques: FlashAttention and Hyena Advances
By
–
There are a few new techniques under review right now that try to get sub-quadratic attention. The most common one I hear about today is FlashAttention, which evolves the technique to increase the context window. Hyena out of Stanford’s Hazy lab explores convolutional approaches.
-
Context Window Limitations Restrict Advanced AI Model Capabilities
By
–
Some of that has to do with restrictions around context windows, which govern how much information can go into and come out of a prompt. Right now those windows generally fall in the 4k to 8k token range. Already that is showing signs of being insufficient for advanced uses.
-
Beyond Transformers: The Next Frontier in AI
By
–
Today’s issue of supervised covers the hunt for what comes after Transformers. That technique, pioneered by Google, has made modern AI possible—but many consider us hitting the “wall” so to speak.
-
Amazon’s Generative AI Job Postings Analysis and Role Organization
By
–
Big analysis of job postings for Amazon on gen Ai here with some awesome details. I also covered some job posting info for gen AI roles (specifically how companies are trying to organize the responsibilities) for the launch of Supervised for the curious! https://
supervised.news/p/generative-a
is-looming-job-title
… -
Strang’s Linear Algebra OCW Course Transforms Math Education
By
–
I'm willing to bet at least 90% of us only completed our math majors because of Strang's OCW linear algebra class.
-
OpenAI’s Secret Sauce Versus Google’s PaLM 2 Performance
By
–
(Though, to be clear, it got very close with its fine-tuned large PaLM 2. So it is also possible that OpenAI has a secret sauce here that Google was not able to match.)