Definition
Expanded definition
Robots.txt can affect what crawlers may access. In AI search work, it is part of technical readiness, but it does not determine how a brand is interpreted once content is accessed. The technical layer matters because AI systems need usable context, but technical access is not the same as good interpretation. Clear entities, current pages, consistent claims, and useful source context still determine whether the answer helps a buyer.
Why it matters
Blocking or allowing important sections can affect source discoverability. Teams should understand what key docs, product pages, and resources are accessible.
Example
A company accidentally blocks a documentation section that explains current product capabilities.
Common mistake
Treating robots.txt as the entire AI readiness strategy.