Question 1

What is a robots.txt file and why do I need one?

Accepted Answer

A robots.txt file is a plain text file placed at the root of your domain that tells web crawlers which parts of your site to access or avoid. It follows the Robots Exclusion Protocol. While not mandatory, it is strongly recommended for any live site with more than a few pages.

Question 2

Does Disallow in robots.txt stop a page from appearing in search results?

Accepted Answer

Not reliably. Disallowing a URL prevents crawlers from visiting it, but Google may still index and display the page if it discovers the URL through external links. Use a noindex meta tag or X-Robots-Tag HTTP header to reliably prevent indexation.

Question 3

What is the difference between Allow and Disallow?

Accepted Answer

Disallow tells a crawler it cannot access a given path. Allow explicitly grants access to a path that would otherwise be blocked by a broader Disallow rule. The more specific rule wins when both apply to the same path.

Robots.txt Generator