I think synthetic data for fine-tuning is a seoarate issue from whether accidental AI-generated text in pre-training data has an impact on model quality
LLMS
-
Flow Engineering with CodiumAI and LangGraph Webinar
By
–
Flow Engineering with CodiumAI & LangChain/LangGraph New Webinar Alert: https://
us06web.zoom.us/webinar/regist
er/WN_fVikSl9eQv68b3ZUdQgwzA#/registration
… "Flow Engineering" is a term that has been gaining in popularity recently. The first time it was mentioned as term was in @CodiumAI paper on AlphaCodium, where they used flow -
Can LLMs perform tasks without external tools?
By
–
Is this something we would expect an LLM to do without tools?
-
Haiku vs GPT-3.5 comparison gains traction over flagship models
By
–
I'm finding the comparison between Haiku and GPT 3.5 a whole lot more interesting than the comparison between Opus and GPT-4
-
Curse of Recursion: 2024 Updates and Vendor Mitigation Strategies
By
–
That's about the "Curse of Recursion" from a year ago – I'm looking for a 2024 update on that. Are there new developments that counter the claims from that paper? What are the big model vendors doing (if anything) to mitigate that risk?
-
AI Models Can Use Links in Prompts for Context
By
–
If the context is online, you can actually paste links to it in the prompt, and the model can decide to use it if it wants! So it’s very likely possible.
-
Planning Pattern: How LLMs Drive Autonomous Agent Decision-Making
By
–
Planning is a key agentic AI design pattern in which we use a large language model (LLM) to autonomously decide on what sequence of steps to execute to accomplish a larger task. For example, if we ask an agent to do online research on a given topic, we might use an LLM to break
-
OpenAI training data transparency and AP licensing concerns
By
–
Yeah I've been wondering about that – is the more recent training data mostly stuff they've licensed from sources like the AP? https://
apnews.com/article/openai
-chatgpt-associated-press-ap-f86f84c5bcc2f3b98074b38521f5f75a
… As always the infuriating lack of training transparency just leaves us guessing -
GPT-4 Predicts Future Events Accurately Through Narrative Storytelling
By
–
Not 100% sure what to make of this timey-wimey paper showing GPT-4 is able to predict the future quite accurately (or, after least make guesses about events that happen after its training cut-off) but only when asked to tell stories about what will happen. https://
arxiv.org/abs/2404.07396 -
Synthetic Data Impact on LLM Pre-training Datasets
By
–
Yeah lots of LLMs are trained on synthetic data these days, especially at the fine-tuning stage – but that's deliberate, the open question for me is still the impact of accidental synthetic data in the larger pre-training data pool