Hm, so that means all weights above values 65504 and below -65504 will be clipped. Does anyone know of there are numbers that large/small in the original bfloat16 representation?
BFloat16 Weight Clipping Limits and Extreme Value Representation
By
–
By
–
Hm, so that means all weights above values 65504 and below -65504 will be clipped. Does anyone know of there are numbers that large/small in the original bfloat16 representation?