I think that's exactly what's happening but for cost reasons rather than accuracy reasons. I think long-context needle-in-the-haystack issues have improved a lot last year since all major LLMs now have a dedicated long-context finetuning stage in pre/post-training.
GENERATIVE AI
-
DeepSeek-R1 Training Cost: Missing R1 Distillation Expenses
By
–
The real question: What is the DeepSeek-R1 training cost? The $5.567M DeepSeek cost is missing the cost of training the R1 model to get distilled data. Table 9 in the DeepSeek v3 paper shows that the R1 distillation step is critical for quality. The R1 paper doesn't talk about
-
Infinite demand for AI tokens: o1 model usage scaling
By
–
If I could run o1 1,000 times on each question I ask I would do it in a heartbeat. There’s infinite demand for tokens.
-
Pre-training remains superior to expensive knowledge injection methods
By
–
Interesting! But knowledge is still best instilled via pre-training. "Relatively" cheap still remains very expensive and cumbersome, even for 7B models, compared to just adding some more embeddings to a database.
-
2025: The Year of LLM Specialization and Multimodal Models
By
–
Remember when we were arguing about the term "foundation models"? It feels like ages ago! With those base models culminating in DeepSeek v3 in Dec 2024, 2025 will likely be the year of LLM specialization! 1. Multimodal LLMs: It was my big prediction for 2024. Most products
-
DeepSeek’s Cost-Efficient Training Methods Explained
By
–
One explanation of how DeepSeek was able to train its model so cheaply..
-
DeepSeek Privacy: US/EU Hosted Open Source Model
By
–
DeepSeek on Perplexity is hosted in US/EU data centers – your data never leaves Western servers. The open source model is hosted completely independent of China. Your privacy and data security is our priority.
-
Document Databases and AI Workloads: Compatibility Myths Debunked
By
–
"Equivalent" . It might be good (and I hope it *is* good in its own right – JSON-native, document-style databases are critical for AI workloads, in particular), but let's not play make believe on compatibility. (Note F's 2.0 blog where they ack perf/etc has been relative)
-
DeepSeek R1 now available on Perplexity Pro Search
By
–
DeepSeek R1 is now available on Perplexity to support deep web research. There's a new Pro Search reasoning mode selector, along with OpenAI o1, with transparent chain of thought into model's reasoning. We're increasing the number of daily uses for both free and paid as add more
-
Runway Academy: Create and Explore Styles with Frames
By
–
Explore existing Styles with Frames or create your own. Learn how in today’s Runway Academy. pic.twitter.com/wqcHgUKUA7
— Runway (@runwayml) 27 janvier 2025Explore existing Styles with Frames or create your own. Learn how in today’s Runway Academy.