A more efficient way to do this could be to have typical logprobs encoded normally and exceptionally low ones marked with a control token as exact magnitudes don’t matter much beyond them being low — this is the approach of SequenceMatch

→ View original post on X — @goodside