Would love to help here. Our crawlers respect robots.txt and support the Crawl-delay directive. X DMs seem to be broken so in the meantime, if can share your server logs at claudebot@anthropic.com, we can try to track down the source or block your domain.
Crawlers Support Robots.txt and Crawl-Delay Directive
By
–
Leave a Reply