@maximelabonne

–

04 July 2024 14h18

Agreed, reward models are great in general. Just not as flexible for some evals.

4 July 2024

Tree Structures for LLM Interaction History Visualization

By

–

03 July 2024 14h59

The tree structure reflects how I interact with these interfaces: I generate a few answers, go back and edit one, etc. It's a lot of back and forth. Even if the LLM can access the full history, I still want trees to visualize my past interactions and to switch between them

LLM Evaluators: Blind Spots and Adversarial Vulnerabilities Exposed

By

–

03 July 2024 14h15

References: – Finding Blind Spots in Evaluator LLMs with Interpretable Checklists https://
arxiv.org/abs/2406.13439
v1
…
– Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment https://
arxiv.org/abs/2402.14016
– On the Limitations of Fine-tuned Judge Models

LLM-as-a-Judge Performance: Key Findings and Model Variability

By

–

03 July 2024 14h10

I reviewed the dark field of LLM-as-a-judge so you don't have to. Here are the key findings. Model Performance Variability
LLMs show inconsistent performance across datasets and tasks. No single model dominates all scenarios. GPT-4 generally leads, with open-source models like

Cannoli installed but cannot find Caret tool

By

–

03 July 2024 13h11

I installed Cannoli but can't find Caret

Obsidian Excellence: Lessons for Chat Interface Design

By

–

03 July 2024 12h41

I do, Obsidian is the best thing. Chat interfaces could learn a lot from it.

LLM Chat Interfaces Need Better UX Design Features

By

–

03 July 2024 12h15

2024: LLM chat interfaces are still ridiculously bad. – There is no tree representation of the current conversation
– You can't edit messages containing artifacts with Claude
– Writing prompts is still 100% manual, and most users are terrible at it

Analyzing LLM Family Trees Through Prompt Genetics

By

–

01 July 2024 16h25

Cool work from @nicolas__yax to analyze LLMs by treating prompts as genes and tokens as alleles. It can recreate models' family trees based only on outputs (e.g., OpenHermes → NeuralHermes). It's particularly nice to get insights into how closed-source models are related.