Update: The “ASCII smuggling” attack first (AFAIK) described in this thread has been applied as part of a high-severity exfiltration exploit chain discovered in Microsoft Copilot, now fixed Nice work, @wunderwuzzi23
!
@goodside
-
ASCII smuggling exploit used to exfiltrate data from Microsoft Copilot
By
–
-
Prompt design: using stop sequences to constrain model output
By
–
Note in the final image:
– Our prompt ends with “Output:” — constraining the model’s response to contain only the output we want
– After our desired response, the base model continues to produce more input/output pairs (only one shown) — avoiding this requires a stop sequence. -
LLM prompting failure fixed with k-shot examples using Claude
By
–
In these screenshots our (illustrative, toy) task is to write a one-line description of a cleaning product. A base LLM will not respect instructions to this effect. The task must be communicated by repeating k-shot examples, which we generate using Claude.
-
Clarifying GPT-3’s WebText training data
By
–
This is historically false. The WebText dataset used (among others) to train GPT-3 consists of the scraped content of URLs linked from Reddit posts with at least 3 karma, including downvotes. You don’t scrape URLs that only appear in low-karma posts because they’re mostly spam.
-
GPT-4 base is not aligned without SFT or RLHF
By
–
That’s why I qualified “aligned” — GPT-4 base isn’t aligned (i.e. no instruct SFT or RLHF).
-
Clarifying what “aligned” means in AI industry
By
–
No, that is what it means. Philosophers of AI safety often mean something bigger, but in industry “aligned” just means “tuned for use” and implies some combination of SFT or RL.
-
Jabberwacky Dialogs Appeared in LLaMA Training Data
By
–
jabberwacky is a pre-LLM chatbot and dialog examples from it appear in llama’s training data so it often appears to assume that context (for the particular chat syntax openrouter adds)
-
Opinion: RLHF vs the ‘Base Model’ Myth
By
–
People hear RLHF and think censorship, so they expect the base model to be untainted, honest, free — all those nice Grokian promises. And it is! But it’s also helplessly insane, and lost in the delusion the entire world is improv theater. “Base model” — not “based model.”
-
LLMs behave like predictive text influenced by chat conditioning
By
–
There’s no dialog before this, to be clear — I didn’t ask it to pretend. LLMs are just like this. Though there’s some some conditioning from the implicit chat syntax (“user:” etc.), to a much greater extent than most widely used LLMs this model really is “just predicting text.”
