For all these results and more, check out the post and let us know your feedback! https://
blog.langchain.dev/extraction-ben
chmarking/
…
AI
-
LangChain Extraction Benchmarking Results and Feedback
By
–
-
Grammar-Based Decoding and Model Quality for Structured Output
By
–
Grammar-based decoding (like logit biasing) helps, but isn't a panacea – model quality is also important for generating correct values, even after you have the structure locked down.
-
LangChain Chat Extraction Benchmarks Documentation Guide
By
–
You can try it yourself on your own setup by following the doc! https://
langchain-ai.github.io/langchain-benc
hmarks/notebooks/extraction/chat_extraction.html
… -
Testing Combined Open Source Strategies for Performance
By
–
Or pick up where we left off testing the open source strategies at this repo. How would it perform if you combine strategies?
-
Testing LLM Performance with Prompting and Decoding Techniques
By
–
We tested popular models like GPT-4, Claude, Llama, and Yi using different prompting and decoding techniques to see how much they impact performance for things like: – Question classification
– Sentiment analysis
– Nested JSON schema correctness -
GPT-4 and Llama Performance Comparison on Tool-Calling Benchmark
By
–
Unsurprisingly, GPT-4 with tool-calling ability performed the best on this benchmark, but Llama, even when not fine-tuned, performed well . See the comparison at this link: https://
smith.langchain.com/public/00f4444
c-9460-4a82-b87a-f50096f1cfef/d/compare?selectedSessions=0c022691-a7ac-4545-b2bc-58aab2d476e8d522dbfc-c09b-45a9-b11e-26aa95a3555a
… -
Prompt Engineering Strategies Impact on Llama Model Performance
By
–
We also explored different prompting strategies using a 32B Llama model to see how they impact output performance. For this task (and model), simple, static strategies didn’t work that well in isolation!
-
LangChain releases new LLM structured extraction benchmark
By
–
Anyone building with LLMs knows structured extraction is key. Today we are releasing a new extraction benchmark testing LLM ability to infer structured data from chat logs. We've also shared some initial results below: https://
blog.langchain.dev/extraction-ben
chmarking/
… -
AI Safety Systems Team Ensures Model Reliability and Societal Benefits
By
–
Our Safety Systems team is on the frontlines of ensuring the safety and reliability of our AI models in the real world today. Learn about the team’s vision, challenges, and structure and see how you can be part of making AI safer and more beneficial for society.
-
Unjustified requirements myths in tech policy and regulation
By
–
And never underrate how frequently “Oh we require that because we have to require it, duh [not actually true]” happens in the world.