not sure what your first question means, but in this case we rely on knowing the tokens of the tokenizer; openAI gives us back key-value pairs of token-logprob
OpenAI Tokenizer: Understanding Token-Logprob Key-Value Pairs
By
–
By
–
not sure what your first question means, but in this case we rely on knowing the tokens of the tokenizer; openAI gives us back key-value pairs of token-logprob