As far as I can tell they usually document their robots.txt policies OpenAI: https://
platform.openai.com/docs/bots Anthropic: https://
support.anthropic.com/en/articles/88
96518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
… Mistral: https://
docs.mistral.ai/robots/

→ View original post on X — @simonw,