A few bits per pair, sure, but so nutritious — lessons not just from humans but from its own inference failures, categorically missing from pre-training. Preferences are some of the only bits a model gets on how to succeed at being itself, with its specific quirks and deficits.