Yes, it’s cheaper and easier, but it’s more of an internal sanity check than outward facing eval to report imho. Btw, spot on regarding including it for the sake of benchmarks. You can tell based on how sensitive some LLMs are to the exact MC prompt format.
PROMPT ENGINEERING
-
Open-ended Evaluations: Free-form Answers and Conversational Performance
By
–
With open-ended evals you mean free-form answers (and measuring conversational performance)?
-
Scaling Parallel Agents for Computer Use Tasks
By
–
what if we stopped betting everything on one agent rollout? "The Unreasonable Effectiveness of Scaling Agents for Computer Use" Generates multiple trajectories in parallel & selects the best using "behavior narratives" 69.9% on OSWorld, nearly matching human-level 72%
-
Verifier and LLM-as-Judge for Output Conformance
By
–
Good suggestions. I'd say those fall into the verifier category (perhaps also LLM-as-a-judge for output-conformance); or do you use something different?
-
Recent Progress in Reasoning Methods Research Synthesis
By
–
It's actually thriving. I probably have bookmarked at >100 interesting papers on reasoning-related methods in just the last few months. I will share some thoughts / synthesis of all this in the upcoming weeks.
-
Open Source Context Engineering Template for AI Agents
By
–
Context Engineering Template for AI Agents. 100% Opensource.
-
Mastering Prompt Engineering Fundamentals
By
–
The meta-lesson from reverse-engineering Anthropic's library:
Prompt engineering isn't about clever tricks. It's about clear communication of: WHO should respond (role)
WHAT they should do (task)
HOW they should do it (process)
WHAT format to use (structure)
WHAT to avoid -
Prefilled responses: underrated prompt technique
By
–
Tenth pattern: they use prefilled assistant responses. This is the most underrated technique in the entire library. Instead of just sending a prompt, Anthropic starts Claude's response.
API structure: { "messages": [ {"role": "user", "content": "Analyze this contract"}, -
Error Handling in AI Prompts
By
–
Ninth discovery: error handling is built into prompts. Anthropic anticipates edge cases and tells Claude how to handle them. If the input data is:
– Incomplete: State what's missing and make reasonable assumptions
– Contradictory: Identify the contradiction and -
Negative examples in prompt engineering
By
–
Sixth technique: constraint specification using negative examples.
Don't just say what you want. Say what you don't want. Standard approach: Write a professional email. Anthropic's method: Write a professional email that:
– Is concise (under 150 words)
– Has a