In both the prompt and response above, each letter in “strawberry” is assigned by the GPT-4o tokenizer to a separate single-character token, instead of being combined into subword-length tokens like “st” “raw” “berry” etc.:
GPT-4o tokenization splits ‘strawberry’ into single-character tokens
By
–
