Yes that's fair, in big O terms the separate QKV should be faster
RESEARCH
-
GPU Precision Evolution: From 32-bit to FP8 in Transformers
By
–
Yes. Originally GPUs were designed for 32 bit precision, but in transformer contexts, you get away with even lower precision (16 bit is the standard usually). In recent months, people pushed that even further to FP8. One of the most recent architectures that successfully did FP8
-
Intellectual Curiosity Behind Parameter Reduction
By
–
just intellectual curiosity reducing the number of params
-

CSAIL Celebrates Valentine’s Day with Midjourney AI Images
By
–
Happy Valentine's Day from CSAIL, w/images from Midjourney.
-
Behind the Scenes: Building Effective Agents Research Post
By
–
Going behind the scenes on one of our latest research posts, Building Effective Agents, with co-authors @ErikSchluntz and @barry_zyj
— Alex Albert (@alexalbert__) 14 février 2025
This was a fun one! We covered a few topics:
– Overrated and underrated agent applications
– Stories that led to them writing the post
– Why… pic.twitter.com/NpmuGtJAcoGoing behind the scenes on one of our latest research posts, Building Effective Agents, with co-authors @ErikSchluntz and @barry_zyj This was a fun one! We covered a few topics:
– Overrated and underrated agent applications
– Stories that led to them writing the post
– Why -
Data Analysis Reveals How Meeting Places Impact Long-Term Relationship Satisfaction
By
–
It's Valentine's Day, and since we love data, we took a look at some data about love! Ever wonder how where a couple meets affects long-term relationship satisfaction? Explore the data to see what real people had to say – and how it’s changed over time.
-
Simplifying Transformer Blocks: Key Research Paper
By
–
Yes. There was a nice paper on that, which looked into some other modifications: Simplifying Transformer Blocks, https://
arxiv.org/abs/2311.01906 -
Drop-in replacement code for modified attention mechanism available
By
–
Yup, the reader posted the code for a drop-in replacement for the modified attention mechanism. Just copy and paste it into the Chapter 5 code, which is available in the GitHub repo linked above.
-

Merging Query and Key Weight Matrices in LLM Training
By
–
Can we merge the query and key weight matrices in an LLM into a single covariance matrix and still train effectively? Here are some promising early results from a reader: https://
github.com/rasbt/LLMs-fro
m-scratch/discussions/517
…
Anyone else familiar with projects that tried this? -
The Future of Transport: AI and Autonomous Vehicles Innovation
By
–
The Future of #Transport
— Ronald van Loon (@Ronald_vanLoon) 14 février 2025
via @YuHelenYu
#AI #ArtificialIntelligence #Transportation #Innovation #AutonomousVehicles
cc: @maxjcm @pascal_bornet @marcusborba pic.twitter.com/KLMRVH497xThe Future of #Transport
via @YuHelenYu #AI #ArtificialIntelligence #Transportation #Innovation #AutonomousVehicles cc: @maxjcm @pascal_bornet @marcusborba