Another one: “Masked ophthalmologists graded the accuracy, relevance, and overall preference [of AI answers]… GPT-4 compared favourably with expert ophthalmologists (median 76%, range 64–90%), ophthalmology trainees (median 59%, range 57–63%), and unspecialised junior doctors…”
RESEARCH
-
Claude 3 Performance Changes: Measurement Challenges and User Perception
By
–
AI performance measurement is ultimately vibes. Lots of people saying Claude 3 got worse. Anthropic says literally nothing major changed. It could be user misperception. It could be that minor change. Or cosmic rays. It could be that Claude subtly reacts to the Ides of April.
-
Dataset Limitations: Missing the Full Distribution of Correct Answers
By
–
Issue in mind is not so much human bias but the fact that the full distribution of correct or desirable answers to your prompts is almost certainly not present in your dataset, only a few samples.
-
Anthropic Reports Claude 3 Quality Drift Update
By
–
Update from the Anthropic team re: Claude 3 quality drift:
-
The Ambiguity Problem in LLM Labeling and Training Data
By
–
Consider being a labeler for an LLM. The prompt is “give me a random number between 1 and 10”. What SFT & RM labels do you contribute? What does this do the network when trained on? In subtle way this problem is present in every prompt that does not have a single unique answer.
-
LLMs Achieve Expert-Level Ophthalmological Knowledge and Reasoning
By
–
"Large language models (LLMs) are approaching expert-level ophthalmological knowledge and reasoning" https://
journals.plos.org/digitalhealth/
article?id=10.1371/journal.pdig.0000341
… @PLOSDigiHealth -
PyTorch DDP Gradient Syncing Bug in Distributed Training
By
–
so last week I was posting about a bug i had with gradient syncing in torch DistributedDataParallel (DDP), aka the most standard way to do multi-GPU training for neural networks all of my misery originated with a single design decision from the torch team: DDP shares
-
Five Years Between Transformer Attention and FlashAttention Innovation
By
–
"5 years between Self-Attention Is All You Need and FlashAttention"
quite incredible stat, gives a pause -
AI for Data Journalism: Practical Applications and Demonstrations
By
–
I gave a 50m talk at the Story Discovery at Scale data journalism conference at Stanford a few weeks ago. The video is now out, and I've written an extensive annotated version AI for Data Journalism: demonstrating what we can do with this stuff right now
-
Better Data Benchmarks Don’t Guarantee Real World AI Impact
By
–
Or, yes, better data matters. But we always have to remember that 'better' is being assessed via narrow benchmarks that DO NOT speak to efficacy or impact in the "real world"