SFT of smaller models will be around for a while I’m sure, but those are smaller models. With each increase in model scale the tradeoff between ICL and SFT tips in favor of the former, and even more so with long context I think.
SYSTEMS
-
VQVAE Stack Architecture Differs Fundamentally from Transformers
By
–
also a “VQVAE stack” is not a transformer! it would not look like this haha
-
Digital Transformation and Industry 4.0 Next Level Model
By
–
A Comprehensive Model of Digital Transformation and the Next Logical Level https://
ow.ly/r7g150QnqfT #digitaltransformation #innovation #iiot #industry40 -
Eos Supercomputer Ranked #9 in TOP500 List
By
–
Ranked #9 in the TOP500 list of fastest supercomputers, Eos is the culmination of our ongoing commitment to pushing the boundaries of #AI technology and infrastructure. Learn more: https://t.co/gJViuPY0Ip #DataCenter #NVIDIADGX pic.twitter.com/AUBfNU7GmR
— NVIDIA (@nvidia) 15 février 2024Ranked #9 in the TOP500 list of fastest supercomputers, Eos is the culmination of our ongoing commitment to pushing the boundaries of #AI technology and infrastructure. Learn more: https://
nvda.ws/3P5dYAf #DataCenter #NVIDIADGX -
Alpha Smoothing Factor Impact on Model Responsiveness
By
–
1. Impact of α (Level Smoothing Factor): Higher α: Impact: A higher α gives more weight to recent observations, making the model more responsive to short-term fluctuations.
-
AI, 5G, IoT, and Cloud Transform Connected Intelligent Devices
By
–
We can envision a world where devices do more than connect—they learn and evolve. #AI is the driving force behind this transformation, with #5G providing the necessary speed, #IoT supplying abundant data, and #Cloud technology ensuring accessibility. #MWC24
-
LLM-Based Multi-Agent Systems: Applications, Benchmarks, and Challenges
By
–
10/ LLM-based Multi-Agents – discusses the essential aspects of LLM-based multi-agent systems; it includes a summary of recent applications for problem-solving and word simulation; summarizes datasets, benchmarks, challenges, and future opportunities.
-
Substrate Limits of Colonizing Consciousness Patterns
By
–
Once you have a colonizing consciousness pattern, the limit of its capabilities are given by the effective size, plasticity, learning rate and determinism of its substrate. Multicellular evolution during which everyone tries to eat your brain is a very slow and noisy way to learn
-
Self-organization learning principles in biological and abiological systems
By
–
Misha Gromov maintains that most of the library of genetic subroutines was discovered quickly and early on in biological evolution. I suspect that the core principle of a learning self organization that imposes itself on matter is either just the cell, or even abiological.
-
Technical trade-offs of LLM inference costs and context window management
By
–
Still WIP I would assume that it would raise cloud costs and load on their systems quite a lot. Not clear also how would it play with the context window itself.