Oh yes it's the nice "Augmenting Self-attention with Persistent Memory" – section 4 – "Feedforward sublayer as an attention layer" 🙂
RESEARCH
-
Simplified Transformer Architecture with Unified Block Design
By
–
TLDR: A much simpler Transformer with a single type of block wired up to a residual pathway in both parallel and in series is possible but to my knowledge has not yet been convincingly demonstrated. Bit more detail @ https://
github.com/karpathy/rando
mfun/blob/master/transformer_unify.ipynb
… -
Transformer Block Unification: MLP and Attention Similarity
By
–
Random quick note on Transformer block unification. People are usually a bit surprised that the MLP and Attention blocks that repeat in a Transformer can be re-formated to look very similar, likely unifiable. The MLP block just attends over data-independent {key: value} nodes:
-
Training Models for Specific Music Styles Using RLHF-Inspired Approaches
By
–
oh i mean split-to-analyse-progress not split-into-different-products, that would be horrible haha i think the insight for pop styles maybe similar to RLHF PPO. instead of generating a bunch of music and seeing what sticks, train models that reflects a style you want & set loose
-
Music Generation Needs Specialization Like DAW Components
By
–
“music generation” is too broad. need to split various jobs of audio out like a DAW would (someone could prob draw an unbundling chart from that) background music generation is essentially now solved. then voice synth, then melodies. feel like pop SHOULD be tractable bc formulaic
-
Few Shot Learning Illustrated Guide and Examples
By
–
Few shot learning, illustrated pic.twitter.com/q0E4Ev9zTx
— swyx 🐣 (@swyx) 28 janvier 2023Few shot learning, illustrated
-
Responsible AI Article Series and Upcoming Robotics Research Coverage
By
–
The latest article in series is Responsible AI. More articles that highlights other areas to come – In particular, excited for robotic one given recent excellent works from Google researchers like Robotic Transformer(RT-1) and SayCan.
-
Google Research 2022: Language, Vision and Generative Models
By
–
Google Research, 2022 & beyond: Language, vision and generative models… A curated series of articles that highlights some of the most impactful AI papers from Google Research in various fields: language, vision, generative models, multimodes, etc. https://
ai.googleblog.com/2023/01/google
-research-2022-beyond-language.html
… -
Remarkable Analysis of ChatGPT Phenomenon by JP_O and Marc Rameaux
By
–
Nulle part, vous trouverez une meilleure analyse du phénomène ChatGPT C’est absolument REMARQUABLE ! A lire d’urgence @JP_O et @Orca_economica (Marc Rameaux) ont fait un boulot fantastique