Glossary · Technical SEO & AI Search

Robots.txt

Definition

Robots.txt is a plain-text file at the root of a website that tells crawlers which URLs they may or may not request.

Also called: Robots exclusion protocol, robots file, REP

Robots.txt lives at a fixed address, such as yourstore.com/robots.txt, and each host (including each subdomain) has its own file. It holds groups of rules. Each group starts with a User-agent line naming a crawler (or * for all crawlers), followed by Disallow and Allow lines for URL paths. Many files also list the XML sitemap. The robots exclusion protocol was formalized as an internet standard, RFC 9309, in 2022.

The file controls crawling, not indexing. Google states that robots.txt is not a mechanism for keeping a page out of Google: a blocked URL can still be indexed if other pages link to it, usually without a description. To keep a page out of search results, use a noindex tag and leave the page crawlable so the crawler can see it. Respectable crawlers follow robots.txt, but others may not, so it is not a security tool.

Why it matters for eCommerce

Stores generate huge numbers of low-value URLs: filter combinations, sort orders, internal search results, cart and account pages. Blocking the worst of them keeps crawlers focused on products and collections, which protects crawl budget on large catalogs. Robots.txt is also where you decide which AI crawlers may fetch your pages, since AI companies publish their own user-agent names.

How it works in practice

On Shopify, the platform generates a default file, and you customize it by adding a robots.txt.liquid template to your theme. On WooCommerce, WordPress serves a virtual file that SEO plugins can edit, or you upload your own. Test changes before publishing, for example with the robots.txt report in Google Search Console.

Common mistakes

  • Leaving “Disallow: /” in place after launching from a staging site.
  • Blocking CSS or JavaScript files that search engines need to render pages.
  • Using robots.txt to remove pages from search results.
  • Blocking a page that carries a noindex tag, so the tag is never seen.

Put the numbers to work

Our free calculators turn these definitions into your numbers, and the Free Profit Audit shows which ones cost you the most.

Free Profit Audit