Yeah, I wouldn't trust that! That does highlight interesting flaw in a lot of open models though: I think there are some models that use strings like [INST] without even reserving a token for them, which opens up all sorts of additional potential prompt injection mischief

→ View original post on X — @simonw