VaultGemma is a release of an open model trained from scratch with differential privacy. The blog post below and the full tech report linked from the tech report have some nice analyses to present a scaling law for differentially private language models: Blog:

→ View original post on X — @jeffdean,