Yes! you can use our SFT notebook available in the model card
MACHINE LEARNING
-
Chain-of-Thought prompting compared to modular code
By
–
The analogy to modular code hit hard. Imagine building an entire app in one function. That’s what we’re doing with CoT.
-
Beyond Benchmarks: Sanmi Koyejo on AI Measurement Science
By
–
Another amazing Keynote at #DLI2025 by @sanmikoyejo, Assistant Professor of Computer Science at @Stanford , Co-Founder of @VirtueAI_co , and President of @black_in_ai, on “Beyond Benchmarks: Building a Science of AI Measurement.” 🧪 Sanmi made a compelling case for moving beyond static benchmarks toward a rigorous science of AI measurement, one that is more reliable, scalable, and meaningful for guiding research, deployment, and policy. This keynote was a powerful reminder that the future of AI depends not just on what we build, but on how we measure it. 📏✨ #urunana #Indaba2025 #AIinAfrica #AIResearch
→ View original post on X — @shakir_za, 2025-08-19 10:22 UTC
-
A 3-Stage Framework for Benchmarking AI Agent Performance
By
–
Want to run your own benchmark? Start with a 3-stage eval: • 1-app tasks debug basic tool calls
• 2- and 3-apps test memory + planning
• Compare long-context vs RAG summaries Log: • Pass rate
• Token usage
• Fail type per task -
5 Design Shifts for Building Better AI Agents
By
–
Want to build better agents? OdysseyBench implies 5 core design shifts: 1. Plan before acting (return explicit step list)
2. Add file/tool validation steps (pre-checks)
3. Use chunked memory, not full transcripts
4. Log task-specific failures (missing file, missing write)
5. -
OdysseyBench: Advanced Metadata for AI Agent Debugging and Evaluation
By
–
OdysseyBench also includes detailed metadata: • Task → App count (1/2/3)
• Token count per session
• Turn-by-turn dialogue history
• Execution plan
• Expected output artifacts Which means… you can analyze why your agent failed, not just if it did. -
GPT-4o Performance Analysis on OdysseyBench-Neo
By
–
Example: GPT-4o on OdysseyBench-Neo: • Long-context: 51.99%, 6.7K tokens
• RAG + chunk summary: 56.29%, 1.36K tokens Semantic compression ≫ brute-force memory More tokens ≠ more understanding. -
Memory design and retrieval strategies in AI models
By
–
Memory design matters more than model size OdysseyBench evaluated multiple retrieval strategies: • Long-context prompting (8K+ tokens)
• RAG with raw session or utterance context
• RAG with summarization: session or chunk Chunk-level summaries outperformed all, with ~75% -
Evaluation Methodology for AI Agent Performance
By
–
Evaluation uses a mix of: • Exact/fuzzy matching on output files
• Execution-based checks (e.g., file created, event logged)
• Pass/fail based on fulfillment of all expected artifacts Every agent output is evaluated against the required plan, not just the final answer.