Robots.txt Tester
Inspect how merutrophy.in controls crawler access, blocked paths, sitemap references and AI crawler rules.
Preview
Score: 76
- At least one user-agent has Disallow: / which blocks the entire site.
- Wildcard (*) patterns should typically be used with explicit paths, e.g. /path/*.
- Important content paths appear blocked in robots.txt.
Get your full report + exact fixes
See what’s hurting your SEO and how to fix it step by step.
- Full breakdown
- Actionable fixes
- Prioritized next steps
Robots.txt Status
Robots.txt Status
Present
Score
76
/100
· Needs review
View Full Robots.txt
Robots.txt Content Preview
# we use Shopify as our ecommerce platform # ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ # ┃ Robots & Agent policy ┃ # ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫ # ┃ Checkouts are for humans. ┃ # ┃ * Automated scraping, “buy-for-me” agents, or any end-to-end flow ┃ # ┃ that completes payment without a final human review step is not ┃ # ┃ permitted. ┃ # ┃ * Legitimate integrators must use the official Checkout Kit: ┃ # ┃ https://www.shopify.com/checkout-kit ┃ # ┃ ┃ # ┃ Terms of Service: https://www.shopify.com/legal/terms ┃ # ┃ Contact: bots@shopify.com ┃ # ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ User-agent: * Disallow: /a/downloads/-/* Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /89379602706/checkouts Disallow: /89379602706/orders Disallow: /carts Disallow: /account Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: */collections/*filter*&*filter* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /policies/ Disallow: /*/policies/ Disallow: /*/*?*ls=*&ls=* Disallow: /*/*?*ls%3D*%3Fls%3D* Disallow: /*/*?*ls%3d*%3fls%3d* Disallow: /search Disallow: /sf_private_access_tokens Disallow: /services/login_with_shop Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail Disallow: /cdn/wpm/*.js Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Sitemap: https://merutrophy.in/sitemap.xml # Google adsbot ignores robots.txt unless specifically named! User-agent: adsbot-google Disallow: /checkouts/ Disallow: /checkout Disallow: /carts Disallow: /orders Disallow: /89379602706/checkouts Disallow: /89379602706/orders Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /cdn/wpm/*.js Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /sf_private_access_tokens Disallow: /services/login_with_shop User-agent: Nutch Disallow: / User-agent: AhrefsBot Crawl-delay: 10 Disallow: /a/downloads/-/* Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /89379602706/checkouts Disallow: /89379602706/orders Disallow: /carts Disallow: /account Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: */collections/*filter*&*filter* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /policies/ Disallow: /*/policies/ Disallow: /*/*?*ls=*&ls=* Disallow: /*/*?*ls%3D*%3Fls%3D* Disallow: /*/*?*ls%3d*%3fls%3d* Disallow: /search Disallow: /sf_private_access_tokens Disallow: /services/login_with_shop Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail Disallow: /cdn/wpm/*.js Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Sitemap: https://merutrophy.in/sitemap.xml User-agent: AhrefsSiteAudit Crawl-delay: 10 Disallow: /a/downloads/-/* Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /89379602706/checkouts Disallow: /89379602706/orders Disallow: /carts Disallow: /account Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: */collections/*filter*&*filter* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /policies/ Disallow: /*/policies/ Disallow: /*/*?*ls=*&ls=* Disallow: /*/*?*ls%3D*%3Fls%3D* Disallow: /*/*?*ls%3d*%3fls%3d* Disallow: /search Disallow: /sf_private_access_tokens Disallow: /services/login_with_shop Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail Disallow: /cdn/wpm/*.js Disallow: /recommendations/products Disallow: /*/recommendations/products Disallow: /products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Disallow: /*/collections/*/products/*-[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]-remote Sitemap: https://merutrophy.in/sitemap.xml User-agent: MJ12bot Crawl-delay: 10 User-agent: Pinterest Crawl-delay: 1 # SEO & AI Discovery Engine - Optimized for search engines and AI bots # Note: Checkout and customer areas remain blocked by Shopify's default Disallow rules # Sitemap directive - CRITICAL for SEO (helps search engines discover all pages) # Allow Googlebot to crawl public content (standard search engine) # Note: Homepage (/) is allowed by default, we only need to explicitly allow public content paths User-agent: Googlebot Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Checkout, cart, orders, account, and admin remain blocked by default Disallow rules # Allow Bingbot to crawl public content (standard search engine) User-agent: Bingbot Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Checkout, cart, orders, account, and admin remain blocked by default Disallow rules # Allow GPTBot (OpenAI) to crawl public product pages User-agent: GPTBot Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Checkout and customer areas remain blocked by default Disallow rules # Allow ChatGPT-User (OpenAI) to crawl public product pages User-agent: ChatGPT-User Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Checkout and customer areas remain blocked by default Disallow rules # Allow anthropic-ai (Anthropic Claude) to crawl public product pages User-agent: anthropic-ai Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Checkout and customer areas remain blocked by default Disallow rules # Allow Claude-Web (Anthropic) to crawl public product pages User-agent: Claude-Web Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Checkout and customer areas remain blocked by default Disallow rules # Allow PerplexityBot (Perplexity AI) to crawl public product pages User-agent: PerplexityBot Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Allow Google-Extended / GoogleOther (Google Gemini) to crawl public product pages User-agent: GoogleOther Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Allow CCBot (Common Crawl) to crawl public product pages User-agent: CCBot Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Allow FacebookBot (Meta AI) to crawl public product pages User-agent: FacebookBot Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Allow YouBot (You.com) to crawl public product pages User-agent: YouBot Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Explicitly allow public content for all crawlers (User-agent: *) # This clarifies that public content is allowed despite the many Disallow rules above # Note: Disallow rules are more specific and still block checkout/customer areas User-agent: * Allow: /products/ Allow: /collections/ Allow: /pages/ Allow: /blogs/ Allow: /llms.txt Allow: /apps/a/llms.txt # Homepage (/) is allowed by default when no explicit Disallow exists # All Disallow rules from above still apply (checkout, cart, orders, account, admin, etc.)
User-agent Rules
| User-agent(s) | Allowed paths | Disallowed paths |
|---|---|---|
| * | No explicit Allow rules. |
|
| adsbot-google | No explicit Allow rules. |
|
| nutch | No explicit Allow rules. |
|
| ahrefsbot | No explicit Allow rules. |
|
| ahrefssiteaudit | No explicit Allow rules. |
|
| mj12bot | No explicit Allow rules. | No explicit Disallow rules. |
| No explicit Allow rules. | No explicit Disallow rules. | |
| googlebot |
|
No explicit Disallow rules. |
| bingbot |
|
No explicit Disallow rules. |
| gptbot |
|
No explicit Disallow rules. |
| chatgpt-user |
|
No explicit Disallow rules. |
| anthropic-ai |
|
No explicit Disallow rules. |
| claude-web |
|
No explicit Disallow rules. |
| perplexitybot |
|
No explicit Disallow rules. |
| googleother |
|
No explicit Disallow rules. |
| ccbot |
|
No explicit Disallow rules. |
| facebookbot |
|
No explicit Disallow rules. |
| youbot |
|
No explicit Disallow rules. |
| * |
|
No explicit Disallow rules. |
Blocked and Allowed Paths
| Blocked paths |
|
|---|---|
| Allowed paths |
|
| Crawl-delay | 10.0 seconds |
Sitemaps Detected
AI Crawler Policy
No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).
Issues Found
- At least one user-agent has Disallow: / which blocks the entire site.
- Wildcard (*) patterns should typically be used with explicit paths, e.g. /path/*.
- Important content paths appear blocked in robots.txt.
Recommendations
- Avoid blocking the entire site (Disallow: /); restrict only sensitive or low-value paths instead.
- Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
- Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.