AI Dynamics

Global AI News Aggregator

Crawlers Support Robots.txt and Crawl-Delay Directive

Would love to help here. Our crawlers respect robots.txt and support the Crawl-delay directive. X DMs seem to be broken so in the meantime, if can share your server logs at claudebot@anthropic.com, we can try to track down the source or block your domain.

→ View original post on X — @alexalbert__,

Commentaires

Leave a Reply

Your email address will not be published. Required fields are marked *