If you are curious how Multimodal LLMs work, I wrote a new article to explain the two main approaches, decoder-only- and cross-attention-style: https://
magazine.sebastianraschka.com/p/understandin
g-multimodal-llms
…
Plus, I reviewed and summarized the 10 latest research papers to see how it's done in practice.
Happy reading!
LLMS
-
Understanding Multimodal LLMs: Decoder-Only and Cross-Attention Approaches
By
–
-
Stanford CS229: Building Large Language Models Explained
By
–
Stanford’s CS229 just dropped an absolute must-watch: Building Large Language Models! In this jam-packed 1.5-hour lecture, they break down how to create models like ChatGPT—from pretraining (language modeling) to post-training (SFT/RLHF). Get the inside track on how #AI
-
Separating LLM Orchestration From Information Processing
By
–
I agree, we should separate out when the LLM is used for orchestrating and when it is used for processing information. The more the two are tied together, the harder is it to debug.
-
OpenAI o1 Leak: Strategic Hype or Accidental Reveal?
By
–
Sneak-peak of o1 In fact, it looks like OpenAI didn't accidentally leak o1 today, but it seems like a well-orchestrated act to create anticipation for what's to come. It's important to remember that Sam Altman has been playing on the hype the whole time. Let's remember how he:
-
Amazon Alexa Delayed as Tech Giants Race for AI Chatbot Dominance
By
–
New Amazon Alexa delayed. While the big tech companies like OpenAI, Anthropic, Microsoft and others are fighting the big battle for the best chatbot, Amazon is still lagging behind. This is irritating, since Amazon was actually one of the first to conquer the market with Alexa!
-
Vinod’s Skepticism of Open Source LLMs and China Concerns
By
–
Vinod is a great guy, but he is (1) quite paranoid about China and (2) heavily invested in OpenAI. So, his skepticism of open source/open weight LLMs is understandable.
-
GPQA Benchmark for Advanced LLM Evaluation
By
–
The GPQA (Graduate-Level Google-Proof Q&A) benchmark is a challenging dataset designed to assess the capabilities of large language models (LLMs) in handling complex, domain-specific questions in biology, physics, and chemistry. It consists of 448 multiple-choice questions
-
o1 reaches level 2 for writing tasks
By
–
o1 has a chance to mark level 2 for writing tasks finally It feels to have much better instruction-following than all previous models. Looking forward to testing it with web search capabilities. Such a model, paired with a big RAG would also make it huge for research!
-
Pliny Jailbreaks OpenAI o1: Latest AI Security Concerns
By
–
Pliny has jailbroken full-o1 within a very short time. He is simply a legend! At the same time, however, it shows how little the models can do against jailbreaking. Pliny repeatedly manages to break through the latest models. As much as I am against doomerism, it does present
-
Model Swarms: Collaborative LLM Expert Algorithm via Swarm Intelligence
By
–
New from @UW and @GoogleDeepMind
: Model Swarms, a collaborative search algorithm that adapts LLM experts to single task, multi-task domains, and reward models via swarm intelligence. Talk to the team @ZifengWang315 @chl260 @YejinChoinka @tsvetshop here! https://
alphaxiv.org/abs/2410.11163
v1
…