Robots.txt lives at a fixed address, such as yourstore.com/robots.txt, and each host (including each subdomain) has its own file. It holds groups of rules. Each group starts with a User-agent line naming a crawler (or * for all crawlers), followed by Disallow and Allow lines for URL paths. Many files also list the XML sitemap. The robots exclusion protocol was formalized as an internet standard, RFC 9309, in 2022.
The file controls crawling, not indexing. Google states that robots.txt is not a mechanism for keeping a page out of Google: a blocked URL can still be indexed if other pages link to it, usually without a description. To keep a page out of search results, use a noindex tag and leave the page crawlable so the crawler can see it. Respectable crawlers follow robots.txt, but others may not, so it is not a security tool.
Why it matters for eCommerce
Stores generate huge numbers of low-value URLs: filter combinations, sort orders, internal search results, cart and account pages. Blocking the worst of them keeps crawlers focused on products and collections, which protects crawl budget on large catalogs. Robots.txt is also where you decide which AI crawlers may fetch your pages, since AI companies publish their own user-agent names.
How it works in practice
On Shopify, the platform generates a default file, and you customize it by adding a robots.txt.liquid template to your theme. On WooCommerce, WordPress serves a virtual file that SEO plugins can edit, or you upload your own. Test changes before publishing, for example with the robots.txt report in Google Search Console.
Common mistakes
- Leaving “Disallow: /” in place after launching from a staging site.
- Blocking CSS or JavaScript files that search engines need to render pages.
- Using robots.txt to remove pages from search results.
- Blocking a page that carries a noindex tag, so the tag is never seen.