Robots.txt is a small text file on your website that tells crawlers which parts they may and may not fetch. It sits in one fixed place, in the root of your domain, and works like a sign at the entrance: it gives directions, but it's not a lock on the door. Search engines read it before they visit your site.
Robots.txt in short
Robots.txt is a simple text file that lets you steer the crawling behaviour of bots. You specify which bots (user-agents) may fetch which paths. That's all it does. It regulates traffic, not access.
- The file is always called
robots.txtand sits in the root of your domain. - You control crawling (fetching), not indexing (being included in the search results).
- It's a public file, so never suitable for keeping something secret.
- Well-behaved bots follow the rules, malicious bots simply ignore them.
- For small sites, a standard file is often enough.
Where is robots.txt located and how does a crawler read the file?
A crawler is a program that fetches pages so it can process them. Before a well-behaved crawler visits your site, it first checks whether there's a robots.txt and what it says. Only then does it start fetching. Think of it as a visitor who reads the sign at the entrance before walking in.
The fixed location of robots.txt
The file must sit in the root of your domain and be named exactly robots.txt For your main domain, that's therefore https://jouwsite.nl/robots.txt. A crawler only looks for the file there. Put it in a subfolder and it won't be found, and simply no rules apply.
Rules don't automatically apply to everything
The scope of robots.txt is limited to exactly that host and that protocol. A file at https://jouwsite.nl/ does not apply to https://shop.jouwsite.nl/ and not to the http variant either. Every subdomain and every protocol therefore needs its own robots.txt. Forget it on your shop subdomain, and that part is technically wide open with no rules.
https://jouwsite.nl/robots.txtapplies to the main domain on https.https://blog.jouwsite.nl/robots.txtis a separate file for the subdomain.- http and https are treated as different sources.
What can you steer with robots.txt?
With robots.txt you decide which bots may fetch which parts of your site. You work with a few simple rules, stacked one below the other. Each rule does one thing, and together they form the instruction for the crawler.
User-agent, Disallow and Allow explained
The three most important rules are User-agent, Disallow en Allow. These determine which bot a rule applies to and which paths it may or may not fetch.
User-agent: which bot the rules apply to. An asterisk (*) means all bots.Disallow: which path the bot may not fetch.Disallow: /prive/excludes that entire folder.Allow: an exception within a blocked path, so you still open up one part.
An empty Disallow: means nothing is blocked and the bot may fetch everything. That way you give clear signals instead of mixed ones. You want the search engine to know exactly where it stands.
“We are in a wild west. Nobody knows exactly how it will play out, so build the foundation that will stand regardless.”
Giacomo Perticara, founder and SEO strategist at GRP Digital
Robots.txt and sitemap
In robots.txt you can also refer to your sitemap. This points crawlers to the list of pages you consider important. It's one of the few positive signals you can give in this file: you're not blocking something, you're pointing the way. Add the full URL of your sitemap.
User-agent: *
Disallow: /prive/
Sitemap: https://jouwsite.nl/sitemap.xmlWhat robots.txt doesn't do
Robots.txt is often overrated. It controls whether a bot fetches a page, and nothing else. It says nothing about indexing and it protects nothing. That distinction is exactly where most misunderstandings start.
Crawling is not the same as indexing
Crawling is fetching a page. Indexing is including it in the search results. Those are two separate steps. If you block a page in robots.txt, Google isn't allowed to fetch it, but it can still end up in the search results if links point to it. Google then shows a bare listing without a description, because it wasn't allowed to read the content. If you actually want to keep a page out of the results, you need a different tool.
Noindex, password protection and other alternatives
If you don't want a page in the search results, use a noindex-instruction instead of robots.txt. Note: that instruction only works if the page is allowed to be crawled, because otherwise Google never reads it. If something really needs to be shielded, put a password on it or handle it at the server level. Those are the only ways to actually lock content down.
- Keeping a page out of the search results: noindex, not robots.txt.
- Actually shielding content: password protection or access control at the server level.
- Steering crawl budget on large sites: robots.txt is the right tool here.
Common mistakes with robots.txt
A mistake in robots.txt is easily made and sometimes goes unnoticed for a long time. Because a single rule already has a big effect, a small error can have major consequences for your visibility. These are the three we run into most often.
Accidentally blocking the entire site
The classic is Disallow: /. That single forward slash shuts your entire site off from crawlers. This often happens by accident when a test environment goes live and the block from that environment comes along with it. So always check what's in your robots.txt after a launch.
Blocking CSS, JavaScript or important resources
Google wants to render your pages the way a visitor sees them. Block the CSS or JavaScript files, and Google sees a broken version of your page and can't assess it as well. Leave these kinds of resources open, even if they seem like side issues.
Using robots.txt for secret content
Robots.txt is a public file that anyone can request via /robots.txt. Put a path in there like Disallow: /geheim-project/, and you're pointing curious visitors straight to exactly what you wanted to hide. Never use it to shield something. That's like a sign saying “don't look inside” on a window with no curtain.
Robots.txt examples for common situations
The best way to understand robots.txt is by looking at examples. Below is a more extensive file and a list of paths you sometimes block in practice.
Simple robots.txt example
This file applies to all bots, blocks a few internal folders, keeps the rest open and points to the sitemap. For many sites, something like this is a fine basis.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /winkelwagen/
Disallow: /bedankt/
Sitemap: https://jouwsite.nl/sitemap.xmlExamples of paths you sometimes block
There's no fixed list that fits every site. What you block depends on your platform and your situation. These paths come up often:
- Internal admin or login pages that have no business being in Google.
- Cart, checkout and thank-you pages of a online shop.
- Internal search result pages that generate endless thin URLs.
- Filter or sort URLs with lots of parameters that repeat the same content.
Never block something just because it's on a list. First check whether the path is genuinely redundant for the search engine before you exclude it.
Robots.txt in SEO: when to use it, when not to
Robots.txt is a useful part of technical SEO, but it's not a button that ranks you higher. It mainly helps you prevent crawlers from wasting time on pages that don't matter. How much you gain from it depends on the size of your site.
For small sites
If you have a small site with a few dozen pages, you barely need to touch robots.txt. Google has no trouble crawling everything. A simple standard file that opens things up and points to your sitemap is usually enough. Don't tinker with this unnecessarily.
For larger or more complex sites
For large online shops or sites with thousands of URLs, robots.txt becomes more interesting. Search engines spend a limited amount of attention on your site, the so-called crawl budget. By excluding unnecessary pages like filter combinations and internal search results, you steer that attention to the pages you actually want in Google. Here, a well thought out robots.txt really pays off.
AI crawlers and robots.txt
Besides search engines, there are bots that fetch content to train LLMs or to feed AI answers. You can address those bots in robots.txt too, via their own user-agent, and specify whether they're allowed to fetch your content.
What this does and doesn't solve
You can politely keep a known AI crawler out by setting its user-agent to Disallow Bots that follow the rules respect that. But robots.txt remains a request, not a lock. A bot that ignores the agreement still fetches your content anyway. If you really want to prevent something from being fetched, a block at the server level is a stronger tool. Think of robots.txt here as a polite request at the front door, not as security.
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /wp-admin/
Sitemap: https://jouwsite.nl/sitemap.xmlFrequently asked questions about robots.txt
What is robots.txt for?
Robots.txt tells crawlers which parts of your site they may and may not fetch. That way you steer their behaviour and prevent them from wasting time on unimportant pages.
What does “blocked by robots.txt” mean?
That message means a page wasn't allowed to be crawled because of a rule in your robots.txt. Often that's intentional, but check whether you really meant to exclude that page.
Does robots.txt actually work?
With well-behaved bots like Google, it works fine. Malicious bots can ignore the rules, so don't rely on it for security.
Does every site need a robots.txt?
Strictly speaking, no: without robots.txt a crawler is simply allowed to fetch everything. Still, a small standard file with a reference to your sitemap is always good practice.
Does robots.txt keep a page out of Google?
No. Robots.txt controls crawling, not indexing. A blocked page can still show up in the results; use a noindex instruction for that.
Can anyone see my robots.txt?
Yes. The file is public and can be requested via jouwsite.nl/robots.txt. So never put paths in it that you'd rather keep hidden.
Conclusion: treat robots.txt as a traffic rule, not a lock
Robots.txt is a simple but powerful file that regulates crawler traffic. It determines what bots may fetch, not what gets indexed, and certainly not what's shielded. Treat it as a sign at the entrance: it points the way for those who follow the rules, but it stops no one who doesn't. If you want to keep a page out of Google, use noindex. If you want to genuinely protect something, put a password on it. Keep the file simple, check it after every launch, and give the search engine clear signals. Then robots.txt does exactly what it's meant to do.
