Most website owners never look at their robots.txt file until something goes wrong. Pages disappear from Google. Crawlers hammer the server. Private folders show up in search results. In many of these cases, a small text file sitting at the root of the site was either missing, misconfigured, or quietly blocking the wrong thing.

A robots.txt file is a plain text file that tells web crawlers which parts of your site they may or may not access. It is one of the oldest and most practical tools in search engine optimization, and it plays a direct role in how search engine bots spend time on your site.

This guide explains what robots.txt is, how it works, and how to use it without accidentally hurting your rankings. You will get real examples, code snippets, common mistakes, and clear best practices.

Learning Objectives

By the end of this article, you will understand:

• The robots txt meaning and what the file actually does
• How a robots txt file communicates with a web crawler
• The protocols behind robots.txt, including the Robots Exclusion Protocol and the Sitemaps protocol
• How User-agent, Disallow, Allow, and Crawl-delay commands work
• How to write a robots txt example for common situations
• The role of robots txt in seo and bot management
• Common mistakes that quietly damage crawling and indexing
• Practical best practices you can apply today

What is robots.txt?

A robots.txt file is a machine-readable file that lives in the root directory of your website. Its job is to give instructions to automated visitors, commonly called web crawlers, search engine bots, or web spiders.

When a crawler arrives at your site, it looks for the file at a fixed location: https://yourdomain.com/robots.txt. The file contains rules that say which URLs the crawler is allowed to request and which it should skip.

Here is the important part. Robots.txt is a set of instructions, not a lock. Well-behaved crawlers read the file and follow it. This is why the distinction between good bots and bad bots matters.

• Good bots, such as Googlebot and Bingbot, respect robots.txt rules.
• Bad bots, such as certain spam bot traffic and scrapers, may ignore the file entirely.

So robots.txt helps you manage cooperative crawler behavior. It does not provide security. If you need to truly restrict access, you need authentication or server-level controls, not a text file.

How does a robots.txt file work?

The process is simpler than it looks. A robots.txt file works through a short request-and-read exchange between the crawler and your web server.

1. A crawler wants to visit your site.
2. Before requesting pages, it fetches yourdomain.com/robots.txt.
3. It reads the rules that apply to its own user agent name.
4. It follows those rules while crawling the rest of the site.

Each rule targets a specific crawler using its user agent string, then lists paths to allow or disallow. If no rule blocks a URL, the crawler treats it as allowed.

It helps to know where this fits technically. Web traffic travels across several layers described by the OSI Model, and robots.txt operates at the application layer as part of the HTTP protocol exchange. You do not need to memorize the OSI Model to use robots.txt, but it is useful to remember that robots.txt is a voluntary agreement carried over standard web requests, not an enforced network rule.

One more detail matters for larger sites. Robots.txt is one input into how search engines manage crawl budget, which is the amount of crawling a search engine is willing to spend on your site. Blocking low-value URLs can help crawlers spend their time on pages that actually matter.

What protocols are used in a robots.txt file?

Two protocols do most of the work.

The Robots Exclusion Protocol. This is the standard that defines how robots.txt is written and interpreted. It sets the syntax for directives like User-agent, Disallow, and Allow. Nearly every major search engine follows it, which is why one file can speak to Google, Bing, Baidu, and many others at once.

The Sitemaps protocol. This is a separate standard that lets you point crawlers to your XML sitemap, a file listing the URLs you want discovered. Robots.txt is a common place to reference your sitemap, so the two protocols often appear together.

Both are open standards. That shared agreement is what allows a single text file to guide crawlers from different companies in a predictable way.

Example of a robots.txt file

A robots.txt example is often clearer than a definition. Here is a straightforward file:

User-agent: *

Disallow: /admin/

Disallow: /cart/

Allow: /

Sitemap: https://www.example.com/sitemap.xml

Reading it line by line:

• User-agent: * applies the rules to all crawlers.
• Disallow: /admin/ blocks the admin directory.
• Disallow: /cart/ blocks the shopping cart pages.
• Allow: / confirms the rest of the site is open.
• Sitemap: points crawlers to the XML sitemap.

Here is a more targeted robots txt example that treats specific crawlers differently:

User-agent: Googlebot

Disallow: /private/

User-agent: Googlebot-Image

Disallow: /images/drafts/

User-agent: Bingbot

Crawl-delay: 10

User-agent: *

Disallow: /tmp/

Sitemap: https://www.example.com/sitemap.xml

This file tells Googlebot to skip /private/, tells Googlebot-Image to avoid draft images, asks Bingbot to slow down, and gives every other crawler a single shared rule.

What is a user agent? What does ‘User-agent: *’ mean?

A user agent is the name a crawler uses to identify itself. When a bot requests a page, it sends this name in its request. Robots.txt uses that same name to decide which rules apply.

Common search engine user agents include:

• Googlebot – Google’s main web crawler
• Googlebot-Image – Google’s image crawler
• Googlebot-News – Google’s news crawler
• Googlebot-Video – Google’s video crawler
• Bingbot – Microsoft Bing’s crawler
• Baiduspider – Baidu’s crawler

You can write rules for each one individually. This is useful when you want, for example, Google robots.txt behavior to differ from how you treat image or video crawling.

The line User-agent: * uses the asterisk as a wildcard. It means “these rules apply to every crawler that does not have its own specific block.” It is the catch-all default.

Keep one rule in mind: if a crawler finds a section written specifically for its name, it follows that section and ignores the * block. So specific rules override the wildcard rule.

How do ‘Disallow’ commands work in a robots.txt file?

The Disallow directive is the core of most robots.txt files. It tells a crawler not to request a specific file or path. The value after Disallow: is a path relative to your root directory.

The mechanics are pattern-based. A crawler compares each URL against your Disallow rules. If the URL path matches a disallowed pattern, the crawler skips it. Let’s look at the four situations you will use most.

Block one file

To keep crawlers away from a single page, name the exact path:

User-agent: *

Disallow: /private-page.html

This blocks only that one file. Everything else stays crawlable. This is a clean approach when a single page, such as a thank-you page or an internal document, should stay out of crawling.

Block one directory

To block an entire folder and everything inside it, end the path with a slash:

User-agent: *

Disallow: /internal/

This tells crawlers to avoid every URL that begins with /internal/. It is a common robots txt disallow pattern for admin areas, staging folders, and internal tools.

Allow full access

Sometimes you want to make it clear that nothing is blocked. You can do that with an empty Disallow value:

User-agent: *

Disallow:

An empty Disallow means “nothing is disallowed,” so the whole site is open to crawling. A missing robots.txt file has a similar practical effect, but stating it explicitly removes doubt.

Hide entire website

To ask crawlers to stay off the entire site, use a single slash. This is the robots txt disallow all pattern:

User-agent: *

Disallow: /

The slash represents the root, so this blocks everything. This setup is common on staging and development sites. It is also one of the most damaging mistakes when it is accidentally pushed to a live site, because it can lead search engines to stop crawling your pages.

A quick caution: blocking a URL in robots.txt does not always keep it out of search results. If other sites link to a blocked page, Google may still list the URL without crawling its content. To reliably keep a page out of the index, use a noindex meta tag on a page that crawlers are allowed to read.

What other commands are part of the Robots Exclusion Protocol?

Beyond User-agent and Disallow, a few more directives round out the Robots Exclusion Protocol.

• Allow – Lets you create an exception inside a disallowed directory. For example, you can block /folder/ but allow /folder/public.html.
• Crawl-delay – Asks a crawler to wait a set number of seconds between requests. This can help reduce load on a busy web server. Google does not use Crawl-delay, but Bing and some others do, so treat it as a request rather than a rule that all crawlers honor.
• Sitemap – Points crawlers to your XML sitemap. It does not restrict anything; it helps with discovery.
• Wildcards – The * matches any sequence of characters, and $ matches the end of a URL. These let you write pattern-based rules, such as Disallow: /*.pdf$ to block all PDF files.

Support for these directives varies between crawlers. Google supports Allow, Sitemap, and wildcard matching, but ignores Crawl-delay. Always confirm behavior for the specific search engines you care about.

What is the Sitemaps protocol? Why is it included in robots.txt?

The Sitemaps protocol is a standard for listing your site’s URLs in a structured, machine-readable file, usually an XML sitemap. That file helps crawlers find pages they might otherwise miss, especially on large sites or sites with pages that are not well linked internally.

Robots.txt and sitemaps solve opposite problems. Robots.txt tells crawlers where not to go. The sitemap tells crawlers where you want them to go. Placing the sitemap reference inside robots.txt gives crawlers both signals in the very first file they read.

Adding your sitemap is simple:

Sitemap: https://www.example.com/sitemap.xml

You can list more than one sitemap line if your site uses several. This does not guarantee indexing, but it does help crawlers discover your important URLs faster.

How does robots.txt relate to SEO and bot management?

This is where robots.txt earns its place in your SEO strategy. Used well, it shapes how crawlers experience your URL structure and where they spend their limited attention.

Here is the practical picture. Search engines allocate a crawl budget to each site. On a large site with thousands of URLs, crawlers can waste time on filter pages, session URLs, and near-duplicate content. Blocking those low-value paths can help crawlers reach your revenue-generating pages more often.

Robots.txt supports several SEO-related goals:

• Managing crawl budget by keeping crawlers away from low-value URLs
• Reducing crawl pressure on your web server during heavy bot traffic
• Steering crawlers toward important sections and your XML sitemap
• Keeping staging and utility paths out of active crawling

Robots.txt is also one layer of bot management, the broader practice of controlling which automated visitors reach your site. It works only for cooperative crawlers, though. The rise of AI crawler bots has made this a live issue, and many site owners now add rules for AI-related user agents. For bad bots that ignore the rules, you need enforcement at the network or firewall level. Services like Cloudflare handle that layer, blocking or challenging traffic that robots.txt alone cannot stop.

So think of robots.txt as guidance for the honest crawlers and a firewall or bot-management service as enforcement for the rest.

[Internal link placeholder: link to a related guide on technical SEO fundamentals.]

Common Mistakes to Avoid

Robots.txt is small, but small errors carry large consequences. These are the mistakes that cause the most damage.

• Blocking the entire site by accident. A leftover Disallow: / from a staging environment is the most common and costly error. Always check this before and after a launch.
• Using robots.txt to hide sensitive data. Disallowed URLs are still visible in the file itself, which anyone can read. Never treat robots.txt as a privacy tool.
• Blocking CSS and JavaScript. If you block the files Google needs to render a page, it may misjudge your layout and mobile usability. Let crawlers access your assets.
• Confusing Disallow with noindex. Disallow stops crawling; it does not remove a page from the index reliably. Use noindex for index control.
• Placing the file in the wrong location. Robots.txt must sit in the root directory. A file at /blog/robots.txt is ignored.
• Forgetting subdomains. Each subdomain needs its own robots.txt. A rule on example.com does not cover shop.example.com.
• Case-sensitivity errors. Paths are case-sensitive. /Folder/ and /folder/ are treated as different.

Read Also: What are the Core Web Vitals (CWV)?

Robots.txt Best Practices

A few habits keep your robots.txt working the way you intend.

1. Keep it simple. Write only the rules you need. Complexity increases the chance of an accidental block.
2. Test before you publish. Use a robots txt tester or robots txt validator to confirm your rules behave as expected. Google Search Console includes a tool to check robots txt against real URLs.
3. Reference your sitemap. Always include your XML sitemap line so crawlers can find your priority URLs.
4. Review after every major change. Redesigns, migrations, and CMS updates can overwrite the file. On WordPress robots txt is often generated dynamically or managed by an SEO plugin, so confirm the output matches your intent.
5. Use a generator when unsure. A reliable robots txt generator can produce correct syntax if you are not comfortable writing rules by hand. Still review the result.
6. Audit it periodically. Run a robots txt checker on a schedule to catch changes you did not expect.
7. Do not block what you want indexed. Before adding any Disallow rule, confirm those URLs are not pages you rely on for organic traffic.

Follow these steps and your file will support crawling and indexing rather than quietly working against it.

FAQs

What is robots.txt used for?

Robots.txt is used to tell web crawlers which parts of your site they may or may not access. It helps manage crawl budget, reduce load from bot traffic during busy periods, keep utility or staging paths out of crawling, and point crawlers to your XML sitemap. It guides crawler behavior; it does not secure or hide content.

Is robots.txt legal?

Yes. Robots.txt is a widely accepted web standard, and using one is standard practice. It is a set of instructions, not a legally binding contract, so it does not by itself force compliance. Whether ignoring robots.txt has legal consequences depends on jurisdiction and the specific activity, such as scraping, rather than on the file itself.

How do I find the robots txt file on my website?

Add /robots.txt to the end of your domain and open it in a browser, for example https://yourdomain.com/robots.txt. If a file exists, you will see its contents as plain text. If you get a 404, you do not have one yet, and you can create a simple text file and upload it to your root directory.

Is robots.txt respected?

Reputable crawlers respect it. Googlebot, Bingbot, Baiduspider, and other major search engine bots follow the Robots Exclusion Protocol. However, bad bots, scrapers, and many spam bots ignore it entirely, because compliance is voluntary. For traffic that does not cooperate, you need firewall rules or a bot-management service rather than robots.txt.

Final Thoughts and Next Steps

Robots.txt is a small file with an outsized effect on how search engines crawl and index your site. When it is written carefully, it helps crawlers spend their time on the pages that matter and keeps clutter out of the way. When it is written carelessly, it can hide pages you need or block resources search engines require. The difference usually comes down to a few lines and a habit of testing.

Your next step is straightforward: open your own /robots.txt, read each line, and confirm every rule reflects what you actually want crawlers to do. Then run it through a validator before you move on.

Need help turning technical SEO into real rankings?

Getting robots.txt right is one piece of a larger picture. If you want a partner to review your crawling setup, fix technical issues, and build a strategy that drives organic traffic, Breakloo Holdings Limited (BHL) can help.

BHL is a leading UK digital agency specializing in SEO, website development, branding, and digital marketing. The team works with businesses to improve Google rankings, strengthen site health, and grow qualified organic traffic through practical, measurable work rather than guesswork.

If you are unsure whether your robots.txt, crawl setup, or wider SEO is helping or holding you back, contact BHL – Breakloo Holdings Limited for a technical SEO review and a clear plan for your next steps.

Breakloo Limited is a full service digital marketing agency and best website development company.