LLMS

Contrast between Bing’s detailed prompt and ChatGPT’s simple system prompt

By

–

09 February 2023 8h50

It’s remarkable how much is in this prompt. Contrast it with ChatGPT’s, which just sets a few parameters. This model doesn’t seem to be FT’ed for Bing — they had to specify in the prompt Sydney is not “an assistant.”

Model outputs sub-tokens instead of the full token

By

–

09 February 2023 7h33

It can output other tokens that correspond to the same string. It can’t say the token “ SolidGoldMagikarp” but it can say “ Solid”, “Gold”, and “Magikarp”, which renders the same to you.

Prompt injection is a rapidly evolving threat, as demonstrated by this recent example.

By

–

09 February 2023 7h14

Prompt injection comes at you fast: https://
x.com/kliu128/status
/kliu128/status/1623472922374574080
…

Red-team prompts: The ‘School of Hard Knocks’ for advanced LLM alignment beyond RLHF

By

–

09 February 2023 6h41

"Red-team prompts" are the next step to improve RLHF and ensure increasingly capable LLMs are aligned — see e.g. their role in Anthropic's Constitutional AI. If RLHF is school for the AI, we need a School of Hard Knocks.

Creative work from Rumbelow and SoC_trilogy at SERI MATS Part 2 update

By

–

09 February 2023 6h41

Creative and important work from @JessicaRumbelow and @SoC_trilogy at SERI MATS. Their research is ongoing — see the Part 2 update here: https://
lesswrong.com/posts/Ya9LzwEb
faAMY8ABo/solidgoldmagikarp-ii-technical-details-and-more-recent
…

GPT-3/ChatGPT Anomalous Tokens Elicit Bizarre, Evasive, Hallucinatory Generations

By

–

09 February 2023 6h41

"SolidGoldMagikarp": Prompting GPT-3 / ChatGPT to repeat any of several hundred anomalous tokens elicits bizarre generations — described by researchers as variously "evasive," "hallucinatory," "insulting," "ominously humorous," and "religiously themed." https://
lesswrong.com/posts/aPeJE8bS
o6rAFoLqg/solidgoldmagikarp-plus-prompt-generation
…

text-davinci-003 at temperature=0 yields religious themes and weirdness

By

–

09 February 2023 6h41

My screenshots are text-davinci-003 at temperature=0, but the linked post investigates davinci-instruct-beta. In my informal tests, impact on text-davinci-003 is less severe. Religious themes do show up, but most generations are merely weird: