I've been using them for a ask a question, get back a SQL query that attempts to answer that question feature I've been building
GENERATIVE AI
-
Local LLMs Weaker Than Hosted Models Like Claude 3
By
–
Most local LLMs are way, way weaker than hosted ones for most purposes – the best I've tried myself has been Mixtral 8x7B but it's still not in the same class as Claude 3 Haiku I haven't run 8x22B locally myself yet, I don't have the RAM for it
-
Choosing Slower Best Models for API Integration in Production
By
–
When building products against APIs – in personal use I still stick to the best available but slowest model for almost everything
-
OpenAI Launches Batch Processing for Workflow Automation
By
–
OpenAI is finally getting into batch data processing, which it turns out is where a lot of actual usage is/will be! Boring, but lots of value in point automation in workflows in the near term. No mythical agents replacing teams, alas!
-
OpenAI Batch Processing Maximizes GPU Utilization Outside Peak Hours
By
–
Last note here though: batch gives you a flexible timeframe which means you can grab idle GPUs outside of peak hours. All the endpoints are mostly in service to maxing out utilization to return the value of those GPUs. So now OpenAI has a way to do that.
-
Unified AI API Services vs Data Privacy Trade-offs
By
–
Advantage here obviously is you're buying one service and get all the APIs—text-to-image/voice, chat completion, etc. But at the same time it's still an API shipping off data vs keeping it where it's hosted. Now they're dealing w/ Snowflake/Databricks on top of other startups.
-
3.5T Batch Pricing: Enterprise-Grade Model Performance Comparison
By
–
It's kind of a unique flavor on batch though for 3.5T, because a) it's more expensive than standard smaller models and b) it's higher performance… so the comparison is more Mixtral-y than Mistral 7B. Maybe enterprise cases evolve to need a Mixtral-grade and not a smaller FT.
-
Choosing AI Models: Speed Cost Trade-offs Beyond Haiku
By
–
For when I'm building user-facing features where speed and cost are important but Haiku isn't quite cutting it
-
LLM Model Scaling: 7B to 8x22B Capability Comparisons
By
–
Yeah I can believe that – but compared to each other I think 7B/8x7B/8x22B may have similar relative characteristics to the differences between Haiku/Sonnet/Opus
-
OpenAI Batch API: Production-Ready Data Processing Solution
By
–
Some notes on the OpenAI batch API. Kind of a better-late-than-never adaptation to what most of the "AI in prod" cases actually look like, which is batch data processing for classic problems like summarization, classification, etc. I talked about this in last Friday's issue.