The thumbs-up/down buttons aren’t for RLHF. The HF in RLHF is selecting the better of two outputs for the same prompt (or, if you have more than two, ordering them best-to-worst). Pointwise data (thumbs, star ratings, etc.) generally doesn’t work well AFAIK.

→ View original post on X — @goodside