English Wikipedia is 29.4B characters, with ~1.5 bits of information per character. So that means, at 3.6 bits per parameter, a 12B model could memorize all english Wikipedia, it seems wild!

→ View original post on X — @aymericroucher