I discussed the challenges of silent data corruption in ML training jobs, and how one faulty piece of hardware can infiltrate and affect the results of a large scale training jobs on thousands of chips.

→ View original post on X — @jeffdean