Here's my bookshelf demo with Gemini
AI
-
Iterative Fine-Tuning Impact on Pre-Training Data Contamination
By
–
How much does deliberately iterative fine-tuning on model outputs tell us about the impact of smaller amounts of generated data making it into larger pre-training datasets?
-
Open Source Library Seeking Evaluation Support
By
–
I do actually have an open source library that needs evals right now – and it's open source (or will be) so no need to hide anything – I'll message you
-
Follow-up Research on Year-Old AI Paper Findings
By
–
That paper is nearly a year old at this point, has there been any follow-on research that supports or refutes it?
-
DSPy: Automatic Prompt Optimization Through Example Derivation
By
–
My current mental model of DSPy is that it leans into the thing where LLMs can produce much better output if you prime them with the right examples – then helps automatically derive the best examples based on running some evals I may have misunderstood it though
-
Base Models Unreliable Self-Assessment Capabilities
By
–
That said, asking a base model directly about its own capabilities feels even less useful to me than asking an instruction-tuned model – models have always been inherently unreliable when it comes to answering questions about themselves
-

MMLU benchmark validity questioned: score improvements may not indicate linear capability gains
By
–
It is, by the way unclear if MMLU is a good measure of anything or if an improvement in scores that is linear is a linear increase in capabilities. That is, going from 89-90 may be a bigger deal than 79-80.
-
RLHF Fine-tuning Effects on Model Behavior Analysis
By
–
That example looks to me like it's more caused by RLHF fine-tuning than anything that was baked into the model in the pre-training phase
-
Are LLMs Really Seeds of Their Own Destruction?
By
–
Right, lots of people really want to believe that LLMs are the inevitable seeds of their own self-destruction – it's a very tempting narrative! I'm trying to understand if it's actually playing out that way
-
RAG and External Sources Impact on AI Models
By
–
I've seen more instant examples of that kind of thing just from RAG using Bing – if someone publishes an article about a model and that article gets into its context at query time via RAG it can have a significant impact