Hello Magento Friends,

It is very important to configure the robots.txt file for your website from the SEO point of view. The robots.txt file contains a set of instructions that guides web crawlers that which page of the website to be indexed and analyzed.

Robots.txt file establishes a relationship between your website and web crawlers. Hence, it is very essential to configure the robots.txt file for Magento 2.

Steps to Configure Robots.txt in Magento 2:

Step 1: From the Admin Panel, go to Content > Design Configuration and select your desired Store.

Step 2: Expand the Search Engine Robots Section. 

robots configuration

  • Choose the option for the Default Robots field. There are several options available,
  • INDEX, FOLLOW – Web crawlers index the page and follow the link on the page.
    NOINDEX, FOLLOW – Web crawlers do not index the page but follow the link on the page.
    INDEX, NOFOLLOW – Web crawlers index the page but do not follow the link.
    NOINDEX, NOFOLLOW – Web crawlers neither index the page nor follow the link on the page.
  • In the Edit Custom Instruction of robots.txt File field, enter custom instructions for web crawlers if any.
  • Reset to Defaults button erases all your robots.txt settings and reset back to default.

Click Save Configuration and flush the cache to reflect the changes.

Learn –  different ways to flush the cache.

Custom Instructions for Magento 2 Robots.txt:

Allows Full Access

Disallows Access to All Folders

Default Instructions

Restrict Checkout and Customer account

Restrict Catalog Search Pages

Restrict Common files

Restrict Common Folders

Restrict Technical Magento Files

Disallow Duplicate Content

Magento 2 Robots.txt Example :

Create robots.txt in Magento root path and add the below code.

NOTE: If your side running in the pub directory so create robots.txt under the pub directory.

Conclusion:

Hence, this way you can Configure Robots.txt in Magento 2. If you need any help mention it in the comment section or reach our support team. Do share the article with your fellow colleagues and stay updated with us!

Happy Reading

Click to rate this post!
[Total: 4 Average: 5]