In my experience, SFT is quite destructive, which is why I like DPO better. There might be a sweet spot with SFT and low LRs though. I haven't experimented with it that much tbh.
RESEARCH
-
LFM Architecture Enables New Foundation Model Design Space
By
–
The LFM architecture opens a new design space for foundation models. This is not restricted to language, but can be applied to other modalities: audio, time series, images, etc. It can also be optimized for specific platforms, like @Apple
, @AMD
, @Qualcomm
, and @cerebras -
LFM-3B Achieves 32K Context Window with Promising RULER Scores
By
–
In this preview release, we focused on delivering the best-in-class 32k context window. These results are extremely promising, but we want to expand it to very, very long contexts. Here are our RULER scores (
https://
github.com/hsiehjackson/R
ULER
…) for LFM-3B ↓ -
LFM optimization outperforms transformers at 1B to 40B scale
By
–
We optimized LFMs to maximize knowledge capacity and multi-step reasoning. As a result, our 1B and 3B models significantly outperform transformer-based models in various benchmarks. And it scales: our 40B MoE (12B activated) is competitive with much bigger dense or MoE models.
-
Three Cancer Care Aspects Ready for AI Enhancement
By
–
3 aspects of cancer care ripe for AI augmentation
#healthcareai #healthtech #FinTech #veuu @gp_pulipaka @stratorob @PetiotEric @EvanKirstel @Fgraillot @HaroldSinnott @HeinzVHoenen @helene_wpli -
What is AI? Exploring Artificial Intelligence and Machine Learning
By
–
What is AI?
#AI #AIio #BigData #ML #NLU #Futureofwork @TopCyberNews @SpirosMargaris @MarshaCollier @MHiesboeck
@MHcommunicate @Fisher85M @MikeQuindazzi @NealSchaffer -
OpenAI Launches Advanced AI Models for Science and Mathematics
By
–
OpenAI has announced the launch of its latest AI models last week, designed to significantly enhance problem-solving capabilities in science, coding, and mathematics. These advanced models boast improved reasoning and critical thinking skills, enabling them to tackle complex
-
Google Researcher Discusses Multimodal Model Design at ECCV
By
–
Google is a proud Diamond Sponsor of the @eccvconf @WiCVworkshop
. Attending the workshop? Hear from Google researcher Viorica Patraucean who works on multimodal model design & evaluation and will give a talk today at 2:45 CEST in the Panorama Lounge! -
Google Gemini 1.5: Deep Dive into New AI Models
By
–
In today's exclusive deep dive with Google & Logan Kilpatrick, we cover: -Google’s new Gemini 1.5 models
-Gemini 1.5 compared to other AI models
-The age of the AI-first developer
-Real-world use cases of Gemini 1.5
-Proactive AI agent systems Read more: http://
therundown.ai/p/an-exclusive
-look-into-googles-new-ai-models
… -
Genetic Algorithms and Evaluation Cost Trade-offs in AI
By
–
Yes I've tried that https://
x.com/maximelabonne/
status/1767124527551549860
… (never trust genetic algorithms when the cost of eval is so high however)