SEO 7 min read
Which AI crawlers to allow and which to block
AI bots do three different jobs: collect training data, fetch pages to ground an answer, and follow a link a user just asked about. Blocking them as one group costs visibility you probably want.