Yeah I think it's a signal that their custom crawler code is full pretty unsophisticated – I would guess it took search engines like Google and Bing years of iteration to get their crawlers to avoid all of the possible dumb mistakes like that

→ View original post on X — @simonw,