Online tools Australia
Robots.txt Generator
A robots.txt file tells well-behaved crawlers which parts of your site to avoid and where to find your sitemap. This generator builds one from your choices, including optional rules for AI training crawlers, and explains what the file does not do. Built for Australian users, with nothing sent to a server.
Free, no signup Runs in your browser
Google ignores crawl-delay; some other crawlers respect it.
Lines
—AI crawlers blocked
—
Sitemaps
—
Copy the output
How do I create a robots.txt file?
Create a plain text file called robots.txt in your site's root folder. Start with User-agent: *, add Disallow lines for the paths you do not want crawled, and add a Sitemap line with the full address of your sitemap. A robots.txt block stops crawling, not indexing, so use a no-index tag to keep a page out of search results.
What to block
Block pages that waste crawl effort and have no search value: internal search results, cart and checkout steps, admin areas and endless filter combinations. Do not block CSS and JavaScript files, since Google needs them to render the page, and do not block pages you want indexed.
A small site rarely needs much more than a Sitemap line.
AI crawlers
Several AI companies publish crawler names and say they honour robots.txt: GPTBot and ChatGPT-User (OpenAI), ClaudeBot and anthropic-ai (Anthropic), CCBot (Common Crawl, used by many AI projects), PerplexityBot, Google-Extended (a control token for Google's AI training) and Applebot-Extended. Blocking them is a policy choice, and compliance is voluntary and may not apply to crawling already done.
To list the pages in your sitemap, check your site's generator or the meta tag analyzer.
How this is calculated
The rule, step by step, so you can check any figure by hand.
User-agent: * applies the rules that follow to every crawlerDisallow: /path asks crawlers not to fetch URLs that start with that pathAllow: /path re-opens a path inside a disallowed folderSitemap: gives the full address of each sitemapnamed user agents, such as GPTBot, get their own block
robots.txt is a request, not a lock. Reputable crawlers follow it, but anyone can read the file and fetch the pages anyway, so never use it to hide private content. The file is also public: listing /secret-admin/ in it announces that folder to everyone.
Blocking crawling does not remove a page from search. If other sites link to a blocked URL, it can still appear in results without a description. To keep a page out, allow crawling and add a no-index tag instead.
A worked example
A WordPress site with a sitemap and AI training crawlers blocked.
- RulesUser-agent: * / Disallow: /wp-admin/ / Allow: /wp-admin/admin-ajax.php
- AI crawlersGPTBot, CCBot, Google-Extended blocked
- Sitemaphttps://www.example.com.au/sitemap.xml
A short file of under twenty lines.
Check it loads at /robots.txt after you upload it.
Where it goes
The file must sit at the root of the host: https://www.example.com.au/robots.txt. Each subdomain needs its own. After uploading, test it in Google Search Console's robots.txt report, which also shows which version Google last fetched.
What it leaves out
Wildcard pattern testing and per-crawler rule sets beyond the AI crawlers are not offered. For complex needs, write the file by hand and test it.
What this calculator does not do
- A request, not security
- Does not remove indexed pages
- Per-crawler rules only for the listed AI crawlers
- No rule tester
Common questions
What does robots.txt do?
It tells crawlers which URLs they should not fetch.
Does robots.txt stop a page being indexed?
No. A blocked page can still appear in results if it is linked elsewhere. Use no-index.
Where do I put robots.txt?
In the root folder of your site, so it loads at /robots.txt.
Should I block AI crawlers?
It is a choice. The listed crawlers say they respect robots.txt, but it is voluntary.
Is robots.txt private?
No. Anyone can read it, so do not list secrets in it.
Where these numbers come from
- Google Search Central — robots.txt What robots.txt does and does not do
- Robots Exclusion Protocol (RFC 9309) The standard for the robots.txt file
- Office of the Australian Information Commissioner Australia's privacy regulator, for the Privacy Act 1988
Method last reviewed 2026-10-28 This calculator uses fixed rules rather than rates that change, and its method is re-checked against the sources above at least once a year. Out of date? Tell us and we will fix it.
Related AU calculators
Meta tag generator
Generate title, description and social meta tags.
Open calculator Online tools.htaccess redirects
Generate .htaccess redirect and HTTPS rules.
Open calculator Online toolsSchema markup generator
Create JSON-LD for a business, FAQ, article or product.
Open calculator Online toolsMeta tags analyzer
Check a page's SEO tags from pasted HTML.
Open calculator