We don't know how to measure LLM abilities well. Most tests are groups of multiple choice questions, tasks, or trivia – they don't represent real world uses well, they are subject to gaming & results are impacted by prompt design in unknown ways. Or they use human preference.
RESEARCH
-
GPT-Fast Integrates Gemma with Optimized Token Performance
By
–
+/-12 lines of code in gpt-fast integrated Gemma from @GoogleDeepMind ; and shows:
* 234 tokens / sec on V100 in int8
* 144 tokens / sec on V100 in float precision https://
github.com/pytorch-labs/g
pt-fast/commit/ef055fc12188eaf80d8ba948ad743ee5583d0f3c
… -
Semantic Chunking: Three Methods for Document Splitting
By
–
Semantic Chunking Semantic chunking using embedding distances between sentences to figure out how to split a document. A key decision is what threshold to split at. We now support three methods: Percentile
Interquartile Range
Standard Deviation Big shout out to -
James Landay Receives ACM SIGCHI Lifetime Research Award
By
–
Congrats to @StanfordHAI
’s James @Landay on his ACM SIGCHI Lifetime Research Award! This award recognizes the most innovative and influential contributions to the study of human-computer interaction. Well deserved, James! @sigchi https://
bit.ly/3Ilw1hH -
Scientific peer review limitations and methodological gatekeeping in research
By
–
"this, but unironically" – scientists
(most fields accept criticism only from within, and are limited by the quality of their peer accepted methodology) -
GTR Model Embeddings Bug Fix and Post-Processing Correction
By
–
unfortunately that's due to a bug I made when training the model; the gtr embeddings i use are missing a last post-processing step. someone trained a fixed model, more info here https://
github.com/jxmorris12/vec
2text/issues/28
… -
ChatGPT’s Lazy Period: Refusing Data Reformatting Tasks
By
–
ChatGPT had a lazy period, where it sometimes would refuse user requests if they looked like too much work. For instance, when asked to reformat a bunch of tabular data into JSON, it would do the first row, and ask the user to do the rest by themselves
-
Tech Engineers vs Ethics Specialists: AI Alignment Leadership Gap
By
–
The tech nerds are mostly infuriated about what is being done to their products. Alignment/ethics/responsibility is done by specialists who usually have degrees in social sciences, but suffer from the de facto demise of the humanities in higher education.
-
Human Preference Predictor Architecture and Results
By
–
This is the architecture of my preference predictor. You can read more about how it works and the results here. https://
huyenchip.com/2024/02/28/pre
dictive-human-preference.html
… As always, feedback is much appreciated! -
Bradley-Terry Model Ranking for AI Model Preference Predictions
By
–
Using the preference predictions for different model pairs for a given prompt, I fit a Bradley-Terry model (the same ranking algorithm that Chatbot Arena uses) to compute a model ranking specific to that prompt.