A dumb crawler doesn't necessarily mean that they're not doing something smarter with the content once it's been crawled – it's generally a good idea to crawl once and then iterate on analysis against your local copy

→ View original post on X — @simonw,