GENERATIVE AI

→ View original post on X — @paulroetzer

GPT-4 Early AGI Version Capabilities According Lex Fridman

By

@paulroetzer

–

16 May 2023 22h09

In the Lex Fridman interview, he seemed to think GPT-4 may be a very early version of AGI. So if there are capabilities they are holding back (which is very likely), it’s possible he believes they are way closer than most would know.

→ View original post on X — @_jasonwei

LLMs and Human Language: A Profound Insight from Stanford Research

By

@_jasonwei

–

16 May 2023 21h54

Clever @google
/
@stanford paper on LLMs from my brother @jerryweiAI
. Performance boosts are great, but there is a more profound insight in this paper that was not explicitly stated: LLMs are trained on human language, but due to the nature of how language was developed (first

→ View original post on X — @rachelmetz

Difficulty Creating ASCII Art, Cow-Like Result

By

@rachelmetz

–

16 May 2023 21h34

I can never get it to make ascii art. At least that kind of looks like a cow?

Together Compute Raises $20M to Overcome Transformer Limitations

By

–

16 May 2023 21h31

I spoke with @togethercompute which is exploring some new techniques to get past the Transformer wall. Together raised a $20m seed from Lux Capital and is also responsible for the RedPajamas open source dataset replicating LLaMA’s set.

MegaByte: Meta’s Latest Context Window Expansion Breakthrough

By

–

16 May 2023 21h29

Finally a third one that is super recent is MegaByte, a paper coming out of Meta last week. This one has been getting a lot of buzz. The three approaches are attempts to expand the context window, which allows for more information coming in (and broader hidden prompts).

Sub-Quadratic Attention Techniques: FlashAttention and Hyena Advances

By

–

16 May 2023 21h28

There are a few new techniques under review right now that try to get sub-quadratic attention. The most common one I hear about today is FlashAttention, which evolves the technique to increase the context window. Hyena out of Stanford’s Hazy lab explores convolutional approaches.

Context Window Limitations Restrict Advanced AI Model Capabilities

By

–

16 May 2023 21h26

Some of that has to do with restrictions around context windows, which govern how much information can go into and come out of a prompt. Right now those windows generally fall in the 4k to 8k token range. Already that is showing signs of being insufficient for advanced uses.

Beyond Transformers: The Next Frontier in AI

By

–

16 May 2023 21h25

Today’s issue of supervised covers the hunt for what comes after Transformers. That technique, pioneered by Google, has made modern AI possible—but many consider us hitting the “wall” so to speak.