Trillion parameter models require terabytes of memory. Thousands of GPUs must be procured and connected just to store the model weights. It takes months to bring up a working cluster of this scale. Nvidia's own chart below:

→ View original post on X — @cerebras