User-agent
Specifies which search engine crawler should follow the rules below it. Use an asterisk to apply instructions to all compliant crawlers.
User-agent: *
Applies to all crawlers.
Control how search engine crawlers access your website. Use our free robots.txt generator to create custom crawl rules, manage access to website directories, and add your XML sitemap without writing code manually.
Create a custom robots.txt file to guide search engine crawlers through your website.
Enter your website and sitemap information.
Choose which crawlers these rules will apply to.
The * user-agent applies to crawlers that don't have a more specific matching group.
Choose common paths to allow or disallow.
These are optional crawl rules, not security controls. Review each path before publishing.
Add your own Allow and Disallow directives.
Download your generated file and upload it to the root directory of your website so it can be accessed at:
https://example.com/robots.txt
A Disallow directive controls crawling, not indexing. Search engines may still show a blocked URL in search results. Do not use robots.txt to protect private pages or confidential files.
A robots.txt file tells search engine crawlers which areas of your website they can or cannot access. It helps guide crawler activity and gives you more control over how search engines explore your site.
User-agent: *Allow: /Disallow: /admin/Disallow: /private/Sitemap: example.com/sitemap.xmlWhen a search engine crawler visits your website, it first checks the robots.txt file in your root directory.
The crawler reads your Allow and Disallow rules to understand which website URLs it is permitted to access.
Compliant crawlers use these instructions when exploring your website, helping you manage crawler access.
You can also include your XML sitemap location to help search engines discover important URLs on your website.
Important SEO note: Robots.txt controls crawler access, not search engine indexing. A blocked URL may still appear in search results if other pages link to it. Use a suitable noindex directive when you need to prevent indexing.
Every robots.txt file uses simple instructions to communicate with search engine crawlers. Learn what each directive does and when to use it.
Specifies which search engine crawler should follow the rules below it. Use an asterisk to apply instructions to all compliant crawlers.
User-agent: *
Applies to all crawlers.
Tells supported crawlers that they may access a specified URL path. It is also useful for allowing a specific path inside a restricted directory.
Allow: /blog/
Allows crawling of the blog directory.
Instructs supported crawlers not to access a specified URL path. It is commonly used for areas that do not need to be crawled.
Disallow: /wp-admin/
Blocks crawling of the WordPress admin directory.
Provides the location of your XML sitemap so search engines can discover your website's important URLs more easily.
Sitemap: https://example.com/sitemap.xml
Points crawlers to your XML sitemap.
Use the generator to combine crawler instructions, manage directories, and add your sitemap in one simple file.
Every website has different crawling requirements. Explore these common configurations and adapt the examples to your website.
Keep important website content accessible while discouraging crawlers from accessing common administrative and internal search paths.
Make sure your important pages and resources remain crawlable. Review your existing rules before replacing them.
User-agent: *
Allow: /
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Sitemap: https://example.com/sitemap.xml
Important: These are starting examples, not universal configurations. Replace example.com with your domain, review every path, and test your rules before publishing. Robots.txt is publicly accessible and should never be used to protect sensitive information.
A small mistake in your robots.txt file can prevent search engines from accessing important pages or resources. Know what to avoid before publishing your file.
Disallowing important service pages, product pages, or blog content can prevent compliant search crawlers from accessing those URLs.
BETTER APPROACH Keep valuable public content crawlable.Robots.txt is publicly accessible. Listing private directories does not protect them from users or malicious crawlers.
BETTER APPROACH Use authentication and proper access controls.Leaving out your sitemap location can make it less convenient for search engines to discover your sitemap through robots.txt.
BETTER APPROACH Add your full sitemap URL.Restricting essential resources can make it harder for search engines to render and understand your pages correctly.
BETTER APPROACH Allow resources needed to render pages.Disallow does not guarantee that a URL will disappear from search results. Crawling and indexing are separate processes.
BETTER APPROACH Use appropriate noindex or access controls.Incorrect paths or overly broad rules can restrict more URLs than intended and create unexpected crawling issues.
BETTER APPROACH Review and test rules before deployment.Creating your robots.txt file is only the first step. Check that your file is accessible, your rules work as intended, and important website pages remain available to search engine crawlers.
Visit your website's robots.txt URL to check whether the file is publicly accessible and contains the expected instructions.
example.com/robots.txt ↗Check each directive against your website structure. Make sure important pages, images, CSS, and JavaScript resources are not unintentionally restricted.
Confirm that your sitemap URL works and contains the correct canonical URLs for your website.
example.com/sitemap.xml ↗Use Search Console reports and URL Inspection to investigate indexing and crawling issues. Check whether important URLs can be crawled by Googlebot.
Open Google Search Console ↗Whenever you update your website structure, plugins, or robots.txt rules, review the file again to catch unintended restrictions.
A quick review can help you avoid accidental crawling restrictions and keep your important website content accessible.
Find answers to common questions about robots.txt files, search engine crawlers, website indexing, and technical SEO.
A robots.txt file is a text file placed in your website's root directory. It provides instructions to compliant search engine crawlers about which URLs they can or cannot access. It is commonly used to manage crawler activity across a website.
Yes. A properly configured robots.txt file can help manage search engine crawling and prevent unnecessary crawler access to certain URL paths. However, it does not directly improve rankings or guarantee that pages will be removed from search results.
Place the robots.txt file in your website's root directory. For example, if your domain is example.com, the file should be accessible at https://example.com/robots.txt. It should use the appropriate protocol and hostname for your website.
No. Disallow prevents compliant crawlers from accessing a matching URL, but the URL may still appear in search results if discovered through other sources. To prevent indexing, use an appropriate noindex directive while allowing crawlers to access the page so they can read it.
No. A website can be crawled without a robots.txt file. However, creating one can be useful when you need to provide crawler instructions or reference your XML sitemap.
Absolutely. WordPress websites can use robots.txt to manage crawler access to selected paths. Review your existing configuration before adding rules, and avoid blocking important pages or resources required to render your website.
Including your XML sitemap URL is a useful way to make its location discoverable to search engines. Use the full absolute URL, such as Sitemap: https://example.com/sitemap.xml.
Yes. You can specify Googlebot as the user-agent and add Disallow rules for the paths you want it to avoid crawling. Be careful with broad rules because they can restrict access to your entire website.
No. Robots.txt is not an access-control or security mechanism. Its instructions are public and may be ignored by non-compliant crawlers. Protect private information with authentication, authorization, and proper server-side security controls.
Review your robots.txt file whenever your website structure, CMS configuration, or crawling requirements change. There is no fixed update schedule; keep it accurate and check important rules after major website changes.
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More