Most local LLMs are way, way weaker than hosted ones for most purposes – the best I've tried myself has been Mixtral 8x7B but it's still not in the same class as Claude 3 Haiku I haven't run 8x22B locally myself yet, I don't have the RAM for it
@simonw
-
Choosing Slower Best Models for API Integration in Production
By
–
When building products against APIs – in personal use I still stick to the best available but slowest model for almost everything
-
Mitigating Prompt Injection: Disable Markdown Images and Links
By
–
While prompt injection remains unsolved, for this particular issue there is a known mitigation: don't ever support markdown (or HTML) images that load images from domains you do not control! More painful but also necessary: don't support outbound clickable links either
-
RAG Security: Prompt Injection Risks in Untrusted Data Processing
By
–
Yet another example illustrating an inconvenient truth: If you build a RAG system that can be used to process data from untrusted sources, and you include the ability to embed images from or links to external domains, then you're going to have prompt injection exfiltration holes
-
Choosing AI Models: Speed Cost Trade-offs Beyond Haiku
By
–
For when I'm building user-facing features where speed and cost are important but Haiku isn't quite cutting it
-
LLM Model Scaling: 7B to 8x22B Capability Comparisons
By
–
Yeah I can believe that – but compared to each other I think 7B/8x7B/8x22B may have similar relative characteristics to the differences between Haiku/Sonnet/Opus
-
Comparing Claude Models: Haiku, Sonnet, and Opus Trade-offs
By
–
Most people (until recently myself included) think of Haiku as the cheap and fast one and Opus as the slow but best one, defining Sonnet is harder because it isn't cheapest or fastest and it isn't best
-
API Billing Limits: Enterprise-Only Monthly Spending Controls
By
–
For a wonderful moment I thought this might let me issue an API key that can only spend a max of x dollars a day, so I can finally ship demos for people to try out without risk of over-spending
— Simon Willison (@simonw) 16 avril 2024
Sadly not: billing limits are per month, and are currently "enterprise only" https://t.co/Pga5JnhYIDFor a wonderful moment I thought this might let me issue an API key that can only spend a max of x dollars a day, so I can finally ship demos for people to try out without risk of over-spending Sadly not: billing limits are per month, and are currently "enterprise only"
-
Switching Between Haiku and Sonnet for Better AI Results
By
–
I've been switching up to Sonnet when Haiku doesn't quite give me good enough results, it's a nice mix of speed and quality
-
Balancing Model Performance and Latency for User-Facing Features
By
–
I'm considering it for user-facing features where latency performance is important but Haiku isn't quite giving me good enough results