Also Agents do honor robots.txt from what i've seen so far

→ View original post on X — @julien_c,