The X-Robots-Tag is an HTTP header that gives search engines indexing instructions for a specific file or page — functioning much like a meta robots tag, but applied at the server level rather than directly within a page's HTML. It's particularly useful for controlling indexing on non-HTML files, like PDFs or images, where a standard meta tag simply can't be inserted.
Common X-Robots-Tag Directives
- noindex — prevents a specific file or page from being indexed at all
- nofollow — tells search engines not to pass ranking value through any links found on that resource
- noarchive — prevents search engines from storing a cached copy
- nosnippet — prevents a text snippet from being shown in search results
Why It's Genuinely Useful for Non-HTML Files
A meta robots tag can only be placed within a page's actual HTML — it has no way to work on a PDF, an image, or a downloadable document. The X-Robots-Tag solves this specific gap by being sent as a server-level HTTP header instead, applying directly to any file type.
Setting It Up
This typically requires direct server configuration, through a site's .htaccess file (on Apache) or its Nginx configuration. Some SEO plugins offer simplified controls for common use cases, but more complex, granular rules generally still require direct server-level configuration.
« Back to Index