"… the communication cost during RingAttention fully overlaps with computation …" Google TPU's fast interconnects are definitely working out well here.
RingAttention Communication Overlaps Computation on Google TPU
By
–
By
–
"… the communication cost during RingAttention fully overlaps with computation …" Google TPU's fast interconnects are definitely working out well here.