Can anyone @lmsysorg confirm if gpt2-chatbot has the ability to run RAG against external tools or if it's working entirely from its own weights? A frustrating thing about opaque model releases is that without knowing details like this they're even harder to evaluate
RESEARCH
-
Models Cannot Accurately Answer Questions About Themselves
By
–
I don't trust models to answer questions about themselves accurately
-
System Prompt Reliability vs Architecture Truthfulness in AI
By
–
It's interesting to me because I trust the system prompt to report a truthful cut-off date more than I trust it to provide a truthful architecture
-
Experiments on GPT2 Chatbot Context Length Limits
By
–
Anyone tried an experiment to see if they can figure out the context length for gpt2-chatbot?
-
New AI Model Development Timeline Testing Strategy
By
–
I would expect a completely new model to have an earlier cutoff date, because there's a bunch of extensive testing you have to do on that new model before you release it even as a preview Trained until Nov then 6 months of internal evaluation before public preview makes sense
-
System Prompts Don’t Guarantee Truthful Model Self-Description
By
–
Worth noting that just because the system prompt says "based on the GPT-4 architecture" doesn't mean that the model is actually based on GPT-4! The goal of a system prompt is to influence the model to behave in certain ways, not to give it truthful information about itself
-
Benchmarking transparency needed for AI model testing
By
–
I'd prefer it if our benchmarking groups were 100% transparent about how their testing stack works and what models they are including in the mix
-
LLM Training Strategies: OpenAI Outputs and Claude Alternatives
By
–
Hmm. Given most LLMs are trained on OAI outputs, those can all share prompts. Then we have Claude, basically can switch out things like “` for . If there's some room for latency, we can train models for this.
-
Extracting AI System Prompts: Challenges and Hallucination Risks
By
–
I'm not sure that's the real system prompt – looks to me like it could be a hallucination based on training data. My own attempts to extract the system prompt have so far failed:
