Hello Magento Friends,
It is very important to configure the robots.txt file for your website from the SEO point of view. The robots.txt file contains a set of instructions that guides web crawlers that which page of the website to be indexed and analyzed.
Robots.txt file establishes a relationship between your website and web crawlers. Hence, it is very essential to configure the robots.txt file for Magento 2.
Contents
Steps to Configure Robots.txt in Magento 2:
Step 1: From the Admin Panel, go to Content > Design Configuration and select your desired Store.
Step 2: Expand the Search Engine Robots Section.
- Choose the option for the Default Robots field. There are several options available,
- INDEX, FOLLOW – Web crawlers index the page and follow the link on the page.
NOINDEX, FOLLOW – Web crawlers do not index the page but follow the link on the page.
INDEX, NOFOLLOW – Web crawlers index the page but do not follow the link.
NOINDEX, NOFOLLOW – Web crawlers neither index the page nor follow the link on the page.
- In the Edit Custom Instruction of robots.txt File field, enter custom instructions for web crawlers if any.
- Reset to Defaults button erases all your robots.txt settings and reset back to default.
Click Save Configuration and flush the cache to reflect the changes.
Learn – different ways to flush the cache.
Custom Instructions for Magento 2 Robots.txt:
Allows Full Access
|
1 2 |
User-agent:* Disallow: |
Disallows Access to All Folders
|
1 2 |
User-agent:* Disallow: / |
Default Instructions
|
1 2 3 4 5 6 7 8 9 10 |
Disallow: /lib/ Disallow: /*.php$ Disallow: /pkginfo/ Disallow: /report/ Disallow: /var/ Disallow: /catalog/ Disallow: /customer/ Disallow: /sendfriend/ Disallow: /review/ Disallow: /*SID= |
Restrict Checkout and Customer account
|
1 2 3 4 5 |
Disallow: /checkout/ Disallow: /onestepcheckout/ Disallow: /customer/ Disallow: /customer/account/ Disallow: /customer/account/login/ |
Restrict Catalog Search Pages
|
1 2 3 |
Disallow: /catalogsearch/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ |
Restrict Common files
|
1 2 3 |
Disallow: /composer.json Disallow: /composer.lock Disallow: /package.json |
Restrict Common Folders
|
1 2 3 4 5 6 |
Disallow: /app/ Disallow: /bin/ Disallow: /dev/ Disallow: /lib/ Disallow: /pub/ Disallow: /phpserver/ |
Restrict Technical Magento Files
|
1 2 3 |
Disallow: / cron.php Disallow: / cron.sh Disallow: / error_log |
Disallow Duplicate Content
|
1 2 |
Disallow: /tag/ Disallow: /review/ |
Magento 2 Robots.txt Example :
Create robots.txt in Magento root path and add the below code.
NOTE: If your side running in the pub directory so create robots.txt under the pub directory.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
# Default Instructions User-agent: * Disallow: /index.php/ Disallow: /*? Disallow: /checkout/ Disallow: /app/ Disallow: /lib/ Disallow: /*.php$ Disallow: /pkginfo/ Disallow: /report/ Disallow: /var/ Disallow: /catalog/ Disallow: /customer/ Disallow: /sendfriend/ Disallow: /review/ Disallow: /*SID= Disallow: /enable-cookies/ Disallow: /LICENSE.txt Disallow: /LICENSE.html # Disable checkout & customer account Disallow: /checkout/ Disallow: /onestepcheckout/ Disallow: /customer/ Disallow: /customer/account/ Disallow: /customer/account/login/ # Restrict Catalog Search Pages Disallow: /catalogsearch/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ # Restrict Common Folders Disallow: /app/ Disallow: /bin/ Disallow: /dev/ Disallow: /lib/ Disallow: /phpserver/ # Disallow Duplicate Content Disallow: /tag/ Disallow: /review/ # Sitemap Sitemap: https://domain.com/sitemap.xml |
Conclusion:
Hence, this way you can Configure Robots.txt in Magento 2. If you need any help mention it in the comment section or reach our support team. Do share the article with your fellow colleagues and stay updated with us!
Happy Reading
