I spoke with @togethercompute which is exploring some new techniques to get past the Transformer wall. Together raised a $20m seed from Lux Capital and is also responsible for the RedPajamas open source dataset replicating LLaMA’s set.
INNOVATION
-
Alteryx Inspire Conference Las Vegas Registration and Top Sessions
By
–
Register at http://
bit.ly/41giHTe for the @Alteryx Inspire Conference in Las Vegas! Top 10 Sessions to Attend: https://
alteryx.com/input/blog/the
-10-top-sessions-to-attend-at-inspire-2023
… — @Alteryx is the Leader in Self-service #DataAnalytics Automation: https://
alteryx.com/alteryx-analyt
ics-automation-platform
…
———
#BigData #DataScience #AI #MachineLearning -
MegaByte: Meta’s Latest Context Window Expansion Breakthrough
By
–
Finally a third one that is super recent is MegaByte, a paper coming out of Meta last week. This one has been getting a lot of buzz. The three approaches are attempts to expand the context window, which allows for more information coming in (and broader hidden prompts).
-
Sub-Quadratic Attention Techniques: FlashAttention and Hyena Advances
By
–
There are a few new techniques under review right now that try to get sub-quadratic attention. The most common one I hear about today is FlashAttention, which evolves the technique to increase the context window. Hyena out of Stanford’s Hazy lab explores convolutional approaches.
-
Context Window Limitations Restrict Advanced AI Model Capabilities
By
–
Some of that has to do with restrictions around context windows, which govern how much information can go into and come out of a prompt. Right now those windows generally fall in the 4k to 8k token range. Already that is showing signs of being insufficient for advanced uses.
-
Beyond Transformers: The Next Frontier in AI
By
–
Today’s issue of supervised covers the hunt for what comes after Transformers. That technique, pioneered by Google, has made modern AI possible—but many consider us hitting the “wall” so to speak.
-
Replit Deployments: Ship Code to Production Faster
By
–
In case you missed it, we recently rolled out Replit Deployments.
— Replit ⠕ (@Replit) 16 mai 2023
Deployments are the quickest way to go from idea to production.
Pair it with Ghostwriter and you’ll be shipping faster than ever before. pic.twitter.com/RR6ygP75spIn case you missed it, we recently rolled out Replit Deployments. Deployments are the quickest way to go from idea to production. Pair it with Ghostwriter and you’ll be shipping faster than ever before.
-
Stability AI Releases StableLM Open Source Language Models
By
–
Stability-AI/StableLM: StableLM: Stability AI Language Models https://
bit.ly/41sGyQ8 #AI #DeepLearning #MachienLearning #DataScience #GenAI -
White House PCAST Generative AI Working Group Invites Public Input
By
–
Excited to be a speaker for the public meeting on #AI on May 19. Info below: #PCAST Working Group on Generative AI Invites Public Input
@WhiteHouse
https://
whitehouse.gov/pcast/briefing
-room/2023/05/13/pcast-working-group-on-generative-ai-invites-public-input/
…