That's about the "Curse of Recursion" from a year ago – I'm looking for a 2024 update on that. Are there new developments that counter the claims from that paper? What are the big model vendors doing (if anything) to mitigate that risk?
@simonw
-
OpenAI training data transparency and AP licensing concerns
By
–
Yeah I've been wondering about that – is the more recent training data mostly stuff they've licensed from sources like the AP? https://
apnews.com/article/openai
-chatgpt-associated-press-ap-f86f84c5bcc2f3b98074b38521f5f75a
… As always the infuriating lack of training transparency just leaves us guessing -
Synthetic Data Impact on LLM Pre-training Datasets
By
–
Yeah lots of LLMs are trained on synthetic data these days, especially at the fine-tuning stage – but that's deliberate, the open question for me is still the impact of accidental synthetic data in the larger pre-training data pool
-
Web crawler code sophistication and search engine iteration
By
–
Yeah I think it's a signal that their custom crawler code is full pretty unsophisticated – I would guess it took search engines like Google and Bing years of iteration to get their crawlers to avoid all of the possible dumb mistakes like that
-
Smart data crawling strategy local iteration analysis
By
–
A dumb crawler doesn't necessarily mean that they're not doing something smarter with the content once it's been crawled – it's generally a good idea to crawl once and then iterate on analysis against your local copy
-
Iterative Fine-Tuning Impact on Pre-Training Data Contamination
By
–
How much does deliberately iterative fine-tuning on model outputs tell us about the impact of smaller amounts of generated data making it into larger pre-training datasets?
-
Open Source Library Seeking Evaluation Support
By
–
I do actually have an open source library that needs evals right now – and it's open source (or will be) so no need to hide anything – I'll message you
-
Follow-up Research on Year-Old AI Paper Findings
By
–
That paper is nearly a year old at this point, has there been any follow-on research that supports or refutes it?
-
DSPy: Automatic Prompt Optimization Through Example Derivation
By
–
My current mental model of DSPy is that it leans into the thing where LLMs can produce much better output if you prime them with the right examples – then helps automatically derive the best examples based on running some evals I may have misunderstood it though