Ooof. I don't know if 128K is possible on the 70B, unless someone can get me a rig with like 16 or more A/H100s. 32K is likely doable though.
70B Model Context Window Limitations and Hardware Requirements
By
–
Global AI News Aggregator
By
–
Ooof. I don't know if 128K is possible on the 70B, unless someone can get me a rig with like 16 or more A/H100s. 32K is likely doable though.
Leave a Reply