Have you explored how this would be affected by low-rank pretraining approaches such as Galore? https://
arxiv.org/abs/2403.03507
Galore Low-Rank Pretraining Impact on Model Training
By
–
By
–
Have you explored how this would be affected by low-rank pretraining approaches such as Galore? https://
arxiv.org/abs/2403.03507