The key distinction: BPE gives us a strong procedure. ConvexTok gives us a procedure, an optimization relaxation, and a certificate. That moves tokenisation from engineering folklore toward measurable optimality.
ConvexTok vs BPE: moving tokenization toward optimality
By
–