I don't think it's absurd to think that from the paper. If I look at Table 1 from https://
arxiv.org/pdf/1906.02243 paper, it sure seems like the cost of the last row of the table is the cost of "Training one model (GPU)". How else should one interpret that line? Not to mention that

→ View original post on X — @jeffdean,