I think it has actually been very unclear. The robots setting prevents crawlers which may include search indexing or other activities from the AI labs. Is there some documentation about this?

→ View original post on X — @emollick,