What robots.txt actually does
A robots.txt file sits at the root of your domain and gives crawlers instructions about where they may fetch pages or files. In practice, it is a crawl-management file. That makes it useful for blocking admin paths, internal search pages, duplicate filter URLs and other areas that do not need repeated crawling.
What it does not do is work as a magic hide button. If a URL is linked elsewhere, it can still be known even when crawling is blocked. That is why you should think of robots.txt as one part of your technical setup, not the whole strategy.
- Keep the file short and readable
- Use it to reduce crawl waste
- Review it after redesigns and migrations
The basic structure
Most files begin with a User-agent line, followed by one or more Allow or Disallow directives. A sitemap line is also common because it tells crawlers where to find the XML sitemap.
For a smaller website, the cleanest version is often the best version. A short file is easier to audit later and much less likely to contain old rules that silently break something.
User-agent: *
Disallow: /admin/
Disallow: /search/
Allow: /admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlExamples for common site types
A small brochure site may only need a sitemap line and one or two disallow rules. A blog might block internal search pages and thin archive variations. A larger ecommerce site may need extra care around faceted navigation, sort parameters and crawler-heavy filter combinations.
The key is to block what clearly creates crawl waste, not to block pages that deserve discovery. Overblocking is one of the most common mistakes on smaller sites.
- Brochure site: keep it minimal
- Blog: block internal search and other thin utility pages
- Store: review filter and sort URLs carefully
Mistakes to avoid
The biggest mistake is trying to do too much. Another one is forgetting that crawlers often need access to CSS and JavaScript resources to render pages properly. A robots.txt file copied from an old forum thread can do more harm than good if it blocks resources or sections that matter.
Another common mistake is leaving old rules in place after a redesign. A file that made sense years ago can quietly block important sections when the structure changes.
- Do not use robots.txt as a replacement for proper indexing control
- Do not block important resources by habit
- Retest the file after structural changes
A simple workflow you can copy
Start by listing which areas of the site actually deserve crawling and which ones only create noise. Draft a short file, keep only the rules that clearly solve a problem and test the result before you upload it.
If you want a fast starting point, use the matching tool to generate a draft, then adjust the lines manually before publishing. That gives you a quicker path without losing human review.
Examples you can adapt
A very small website might only need a sitemap line and one or two disallow rules. A larger content site may need to block internal search, session-based URLs or low-value utility paths that attract crawler traffic without helping users.
The safest way to work is to start narrow. Block only what is obviously unnecessary, then review the file after a crawl or after the site structure changes. That approach is usually better than copying a long robots.txt example from another website with completely different needs.
- Keep examples specific to the type of site you actually run
- Document why each rule exists so old lines can be removed later
- Retest after migrations, redesigns and plugin changes
A practical checklist before you publish
Before uploading the file, check that important pages, templates and resources are still crawlable. If you are working on a website with multiple environments, make sure staging rules do not accidentally reach production.
It also helps to compare the finished file with your sitemap and internal links. If the sitemap invites crawlers into sections that robots.txt blocks, the setup is usually telling mixed stories.
- Review the live paths one by one
- Check that CSS and JavaScript resources are still reachable
- Make sure the sitemap line points to the correct domain
Why robots.txt still matters on modern websites
Robots.txt is not the most glamorous file on a website, but it still plays a practical role in keeping crawlers focused on the pages that matter most. On small sites, it can be the difference between a clean crawl path and a crawler spending time on internal search pages, utility folders, preview URLs or old parameter-based paths that add no SEO value.
On larger websites, the file becomes even more useful because crawl waste usually grows with the platform. Filters, sorting options, session-based URLs, faceted navigation and duplicated query strings can quickly multiply the number of URLs a crawler can request. A short robots.txt file will not solve every crawling issue, but it is often the first layer of defence.
The safest mindset is to keep the file practical and minimal. Only add a rule if you clearly understand what it blocks, why the blocked section is low value, and what effect the rule should have on the way the site is crawled.
- Use robots.txt to guide crawling, not to hide sensitive data
- Treat every new disallow rule as a technical change worth reviewing
- Revisit the file after migrations, redesigns and major plugin installs
When you should and should not block a path
A good block candidate is a section that creates crawler demand without helping real search visibility. Internal search pages are the classic example. So are thin faceted URLs with almost no unique value, temporary test folders and admin paths that no public user needs to discover.
A poor block candidate is a section that contains assets or pages required to understand the website properly. If a template needs CSS or JavaScript to render a page layout, blocking those files can make pages harder to process. If a category page is meant to rank, it should not be hidden just because it looks similar to another page.
When in doubt, ask a simpler question: does this path exist to serve users, or does it mainly exist because of the way the system is built? If it is mostly a system by-product, it is a stronger candidate for a robots rule.
A safer review process before publishing changes
The best review process is boring, and that is a compliment. Draft the file, compare it with the current site structure, check the folders and URLs one by one, then upload it only after a final read-through. A rushed copy-and-paste is how most robots.txt mistakes happen.
It also helps to keep a short changelog. Even a note like blocked internal search pages after launch is enough to explain why a rule exists. Six months later, that note can save you from deleting a useful line or keeping a harmful one by accident.
If you maintain multiple environments, create separate files or deploy rules carefully. One of the oldest and easiest mistakes is accidentally carrying a staging block into production.
Practical examples for common situations
A local business site usually needs very little: perhaps one admin path blocked and a sitemap line. A content site may also block internal search or tag combinations if they are generating thin, repetitive pages. An ecommerce site often needs the most care because layered navigation and filtered views can create a large number of machine-generated URLs.
This is where a generator is useful. It gives you a clean draft quickly, but the real value comes from reviewing the draft in the context of your own site. The right robots.txt file for a six-page brochure site is not the same as the right file for a store with thousands of products.
Whenever a rule feels too broad, it probably is. Narrower rules are easier to trust and easier to undo.
User-agent: *
Disallow: /search/
Disallow: /preview/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlWhat to do after you upload the file
Publishing the file is not the end of the task. After the upload, check that the file loads correctly at the root of the domain, then review the most important sections of the site to make sure nothing valuable is affected. That quick check catches a surprising number of mistakes.
It is also worth revisiting your internal links, sitemap and canonical setup. Technical signals work best when they tell the same story. A robots.txt file that blocks a folder while the sitemap advertises the same URLs is usually a sign that the overall setup needs another pass.
As the website grows, the goal should remain the same: keep the file easy to understand. A robots.txt file that nobody wants to touch is often a file that nobody fully trusts.
A practical checklist you can use right away
If you want the shortest path from theory to action, turn the advice in this guide into a small checklist you can repeat on every page or project. That is usually the difference between understanding an idea once and using it consistently.
A useful checklist should be short enough to follow and specific enough to catch mistakes. In practice, that means listing the decisions that matter most, reviewing a real example, then applying the same sequence the next time you touch the same type of task.
For most site owners, repeatable process beats perfect memory. The more technical or repetitive the task is, the more valuable that becomes.
- Review the current setup before changing anything
- Make one clear draft instead of stacking half-finished edits
- Check the result on the live site after publishing
Examples that show the idea in context
Examples matter because abstract advice can sound obvious until you have to apply it to a real page. A blog article, a store category, a tool landing page and a documentation page often need slightly different decisions even when the overall principle stays the same.
That is why it helps to compare at least two or three realistic scenarios whenever you implement the ideas from this guide. The principle stays stable, but the execution changes depending on the page type and the goal.
When in doubt, work from the actual page purpose first. The right implementation nearly always becomes clearer once the page job is obvious.