---
title: "How to Configure Robots.txt in Magento 2?"
url: "https://magecomp.com/blog/configure-robots-txt-magento-2/"
date: "2021-09-16T10:00:41+00:00"
modified: "2021-09-16T10:00:41+00:00"
author:
  name: "Dhiren Vasoya"
  url: "https://magecomp.com/"
categories:
  - "Magento Tutorials"
word_count: 524
reading_time: "3 min read"
summary: "Hello Magento Friends,"
description: "Robots.txt file help web crawlers know which page of the website to be indexed. Learn How to Configure Robots.txt in Magento 2?"
keywords: "Magento Tutorials"
language: "en"
schema_type: "Article"
related_posts:
  - title: "What’s the difference between Allowed Countries, EU Countries, and Top Destinations in Magento?"
    url: "https://magecomp.com/blog/difference-between-allowed-eu-and-top-countries-in-magento/"
  - title: "Procedure for Setting the Locale, Language, and Country in Magento 2?"
    url: "https://magecomp.com/blog/set-locale-language-country-magento-2/"
  - title: "How to Enable Automatic Backup in Magento 2"
    url: "https://magecomp.com/blog/enable-automatic-backup-magento-2/"
---

# How to Configure Robots.txt in Magento 2?

_Published: September 16, 2021_  
_Author: Dhiren Vasoya_  

![How to Configure Robots.txt in Magento 2](https://magecomp.com/blog/wp-content/uploads/2021/09/How-to-Configure-Robots.txt-in-Magento-2-1024x512.webp)

**Hello Magento Friends,**

It is very important to configure the robots.txt file for your website from the SEO point of view. The robots.txt file contains a set of instructions that guides web crawlers that which page of the website to be indexed and analyzed.
Robots.txt file establishes a relationship between your website and web crawlers. Hence, it is very essential to configure the robots.txt file for Magento 2.
## Steps to Configure Robots.txt in Magento 2:
**Step 1:** From the Admin Panel, go to **Content > Design Configuration** and select your desired **Store**.
**Step 2:** Expand the **Search Engine Robots** Section.
[![robots configuration](https://magecomp.com/blog/wp-content/uploads/2021/09/robots-configuration.png)](https://magecomp.com/blog/wp-content/uploads/2021/09/robots-configuration.png)

- Choose the option for the **Default Robots** field. There are several options available,- **INDEX, FOLLOW** – Web crawlers index the page and follow the link on the page.
    **NOINDEX, FOLLOW** – Web crawlers do not index the page but follow the link on the page.
    **INDEX, NOFOLLOW** – Web crawlers index the page but do not follow the link.
    **NOINDEX, NOFOLLOW** – Web crawlers neither index the page nor follow the link on the page.
    - In the **Edit Custom Instruction of robots.txt File** field, enter custom instructions for web crawlers if any.- **Reset to Defaults** button erases all your robots.txt settings and reset back to default.
Click **Save Configuration** and flush the cache to reflect the changes.
Learn – [**different ways to flush the cache**](https://magecomp.com/blog/magento-2-difference-between-cache-clean-cache-flush/).
## Custom Instructions for Magento 2 Robots.txt:
**Allows Full Access**

```
User-agent:*
Disallow:
```

**Disallows Access to All Folders**

```
User-agent:*
Disallow: /
```

**Default Instructions**

```
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
```

**Restrict Checkout and Customer account**

```
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/
```

**Restrict Catalog Search Pages**

```
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
```

**Restrict Common files**

```
Disallow: /composer.json
Disallow: /composer.lock
Disallow: /package.json
```

**Restrict Common Folders**

```
Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /pub/
Disallow: /phpserver/
```

**Restrict Technical Magento Files**

```
Disallow: / cron.php
Disallow: / cron.sh
Disallow: / error_log
```

**Disallow Duplicate Content**

```
Disallow: /tag/
Disallow: /review/
```

## Magento 2 Robots.txt Example :
Create robots.txt in Magento root path and add the below code.
**NOTE:** If your side running in the pub directory so create robots.txt under the pub directory.
```
# Default Instructions
User-agent: *
Disallow: /index.php/
Disallow: /*?
Disallow: /checkout/
Disallow: /app/
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
Disallow: /enable-cookies/
Disallow: /LICENSE.txt
Disallow: /LICENSE.html

# Disable checkout & customer account
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/

# Restrict Catalog Search Pages
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/

# Restrict Common Folders
Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/

# Disallow Duplicate Content
Disallow: /tag/
Disallow: /review/

# Sitemap
Sitemap: https://domain.com/sitemap.xml
```

## Conclusion:
Hence, this way you can Configure Robots.txt in Magento 2. If you need any help mention it in the comment section or reach our [**support team**](https://magecomp.com/support/). Do share the article with your fellow colleagues and stay updated with us!
**Happy Reading**


---

_View the original post at: [https://magecomp.com/blog/configure-robots-txt-magento-2/](https://magecomp.com/blog/configure-robots-txt-magento-2/)_  
_Served as markdown by [Third Audience](https://github.com/third-audience) v3.5.3_  
_Generated: 2026-08-10 02:49:01 UTC_