Robots.txt is a simple text file placed at the root of a website, telling search engine crawlers which parts of the site they should or shouldn't crawl. It's one of the more foundational elements of technical SEO.
What Robots.txt Actually Controls
- Which specific directories or pages search engines are allowed to crawl
- Which areas are explicitly blocked from crawling (like an admin login page)
- The location of a site's XML sitemap
A Simple Example
A rule like `Disallow: /wp-admin/` tells search engines not to crawl a WordPress site's admin area — content that, quite sensibly, serves no purpose appearing in public search results.
Common Mistakes to Avoid
- Accidentally blocking a site's entire content from being crawled at all
- Assuming robots.txt actually hides sensitive pages — it doesn't; it only requests that crawlers not visit them
- Forgetting to properly reference the sitemap's location within the file
Managing Robots.txt in WordPress
WordPress generates a basic, sensible robots.txt file automatically, and SEO plugins like Yoast or Rank Math offer a simple, direct editor for customizing it further, without needing separate FTP access to edit the file manually.
« Back to Index