Robots.txt Tester
Inspect how amazon.com controls crawler access, blocked paths, sitemap references and AI crawler rules.
Robots.txt Status
Robots.txt Status
Present
Score
68
/100
ยท Needs review
View Full Robots.txt
Robots.txt Content Preview
User-agent: * Disallow: /exec/obidos/account-access-login Disallow: /exec/obidos/change-style Disallow: /exec/obidos/flex-sign-in Disallow: /exec/obidos/handle-buy-box Disallow: /exec/obidos/tg/cm/member/ Disallow: /gp/aw/help/id=sss Disallow: /gp/cart Disallow: /gp/flex Disallow: /gp/product/e-mail-friend Disallow: /gp/product/product-availability Disallow: /gp/product/rate-this-item Disallow: /gp/sign-in Disallow: /gp/reader Disallow: /gp/sitbv3/reader Disallow: /gp/richpub/syltguides/create Disallow: /gp/gfix Disallow: /gp/associations/wizard.html Disallow: /gp/dmusic/order Disallow: /gp/legacy-handle-buy-box.html Disallow: /gp/aws/ssop Disallow: /gp/yourstore Disallow: /gp/gift-central/organizer/add-wishlist Disallow: /gp/vote Disallow: /gp/voting/ Disallow: /gp/music/wma-pop-up Disallow: /gp/customer-images Disallow: /gp/richpub/listmania/createpipeline Disallow: /gp/content-form Disallow: /gp/pdp/invitation/invite Disallow: /gp/customer-reviews/common/du Disallow: /gp/customer-reviews/write-a-review.html Disallow: /gp/associations/wizard.html Disallow: /gp/music/clipserve Disallow: /gp/customer-media/upload Disallow: /gp/history Disallow: /gp/item-dispatch Disallow: /gp/dmusic/order/handle-buy-box.html Disallow: /gp/recsradio Disallow: /gp/slredirect Disallow: /dp/shipping/ Disallow: /dp/twister-update/ Disallow: /dp/manual-submit/ Disallow: /dp/e-mail-friend/ Disallow: /dp/product-availability/ Disallow: /dp/rate-this-item/ Disallow: /gp/registry/wishlist/*/reserve Disallow: /gp/structured-ratings/actions/get-experience.html Disallow: /gp/twitter/ Disallow: /ap/signin Disallow: /gp/registry/wishlist/ Disallow: /wishlist/ Allow: /wishlist/universal* Allow: /wishlist/vendor-button* Allow: /wishlist/get-button* Disallow: /gp/wishlist/ Allow: /gp/wishlist/universal* Allow: /gp/wishlist/vendor-button* Allow: /gp/wishlist/ipad-install* Disallow: /registry/wishlist/ Disallow: /review/common/du Disallow: /gp/registry/search.html Disallow: /product-reviews/B0069IY63Y Disallow: /gp/orc/rml/ Disallow: */gcrnsts Disallow: /gp/gc/widget Disallow: /gp/dmusic/mp3/player Disallow: /gp/entity-alert/external Disallow: /gp/customer-reviews/dynamic/sims-box Disallow: /review/dynamic/sims-box Disallow: /gp/redirect.html Disallow: /gp/twister/ajaxv2 Disallow: /ss/twister/ajax Disallow: /b?*node=7454917011 Disallow: /b?*node=7454927011 Disallow: /b?*node=7454939011 Disallow: /b?*node=7454898011 Disallow: /gp/customer-media/actions/delete/ Disallow: /gp/customer-media/actions/edit-caption/ Disallow: /gp/dmusic/ Allow: /gp/dmusic/promotions/PrimeMusic Allow: /gp/dmusic/promotions/AmazonMusicUnlimited Disallow: /gp/offer-listing/ Disallow: /b?*node=9052533011 Disallow: /lm/R1XIHQVKXSKBNJ Disallow: /lm/R3HQ5WJSZK6QSO Disallow: /surprise/ Disallow: /local/ajax/ Disallow: /gp/socialmedia/giveaways Disallow: /gp/b2b-rd Disallow: /gp/aw/so.html Disallow: /reviews/iframe Disallow: /gp/switch-language Disallow: /ga/p/ Disallow: /gp/profile/ Disallow: /giveaway/host/setup/ Disallow: /ss/customer-reviews/lighthouse/ Disallow: /ospublishing/story/* Disallow: /gp/aw/ol/ Disallow: /gp/promotion/ Disallow: /hz/leaderboard/top-reviewers/ Disallow: /creatorhub Disallow: /creatorhub/* Disallow: /slp/s$ Disallow: /-/ Allow: /-/es/ Allow: /-/en$ Allow: /-/zh_TW/ Allow: /-/zh_TW$ Allow: /-/he/ Allow: /-/he$ Disallow: /hz/help/contact/*/message/$ Disallow: /gp/aw/shoppingAids/ Disallow: /rss/people/*/reviews Disallow: /gp/pdp/rss/*/reviews Disallow: /gp/cdp/member-reviews/ Disallow: /gp/aw/cr/ Allow: /gp/offer-listing/B000 Allow: /gp/offer-listing/9000 Disallow: /gp/aag Allow: /gp/aag/main?*seller=ABVFEJU8LS620 Disallow: /gp/pdp/profile/ Disallow: /gp/help/customer/express/c2c/ Disallow: /slp/*/b$ Disallow: /hz/contact-us/ajax/initiate-trusted-contact/ Disallow: /hp/video/ Disallow: /gp/video/api Disallow: /gp/video/settings Disallow: /gp/video/library Disallow: /gp/video/watchlist Disallow: /gp/video/mystuff Disallow: /gp/video/profiles Disallow: /gp/video/search Disallow: /gp/video/auth Disallow: /gp/video/salp/i Disallow: /icdp/ User-agent: EtaoSpider Disallow: / User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-Agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GoogleAgent-Mariner Disallow: / User-agent: GoogleAgent-Shopping Disallow: / User-agent: ClaudeBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Bytespider Disallow: / User-agent: Scrapy Disallow: / User-agent: PetalBot Disallow: / User-agent: Devin Disallow: / User-agent: omgili Disallow: / User-agent: AI2Bot Disallow: / User-agent: Gemini-Deep-Research Disallow: / User-agent: PanguBot Disallow: / User-agent: MistralAI-User Disallow: / User-agent: Diffbot Disallow: / User-agent: Sidetrade indexer bot Disallow: / User-agent: Google-NotebookLM Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: Copilot Disallow: / User-agent: Ai2Bot-Dolma Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: meta-externalfetcher Disallow: / User-agent: iaskspider Disallow: / User-agent: wpbot Disallow: / User-agent: aiHitBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: img2dataset Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: YouBot Disallow: / User-agent: Brightbot Disallow: / User-agent: Claude-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: ISSCyberRiskCrawler Disallow: / User-agent: CopilotNative Disallow: / User-agent: CopilotSapphire Disallow: / User-agent: imgproxy Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: Timpibot Disallow: / User-agent: Thinkbot Disallow: /
User-agent Rules
| User-agent(s) | Allowed paths | Disallowed paths |
|---|---|---|
| * |
|
|
| etaospider | No explicit Allow rules. |
|
| gptbot | No explicit Allow rules. |
|
| ccbot | No explicit Allow rules. |
|
| perplexitybot | No explicit Allow rules. |
|
| google-extended | No explicit Allow rules. |
|
| googleagent-mariner | No explicit Allow rules. |
|
| googleagent-shopping | No explicit Allow rules. |
|
| claudebot | No explicit Allow rules. |
|
| meta-externalagent | No explicit Allow rules. |
|
| bytespider | No explicit Allow rules. |
|
| scrapy | No explicit Allow rules. |
|
| petalbot | No explicit Allow rules. |
|
| devin | No explicit Allow rules. |
|
| omgili | No explicit Allow rules. |
|
| ai2bot | No explicit Allow rules. |
|
| gemini-deep-research | No explicit Allow rules. |
|
| pangubot | No explicit Allow rules. |
|
| mistralai-user | No explicit Allow rules. |
|
| diffbot | No explicit Allow rules. |
|
| sidetrade indexer bot | No explicit Allow rules. |
|
| google-notebooklm | No explicit Allow rules. |
|
| duckassistbot | No explicit Allow rules. |
|
| copilot | No explicit Allow rules. |
|
| ai2bot-dolma | No explicit Allow rules. |
|
| google-cloudvertexbot | No explicit Allow rules. |
|
| meta-externalfetcher | No explicit Allow rules. |
|
| iaskspider | No explicit Allow rules. |
|
| wpbot | No explicit Allow rules. |
|
| aihitbot | No explicit Allow rules. |
|
| cohere-ai | No explicit Allow rules. |
|
| friendlycrawler | No explicit Allow rules. |
|
| img2dataset | No explicit Allow rules. |
|
| velenpublicwebcrawler | No explicit Allow rules. |
|
| youbot | No explicit Allow rules. |
|
| brightbot | No explicit Allow rules. |
|
| claude-user | No explicit Allow rules. |
|
| claudebot | No explicit Allow rules. |
|
| claude-searchbot | No explicit Allow rules. |
|
| perplexity-user | No explicit Allow rules. |
|
| isscyberriskcrawler | No explicit Allow rules. |
|
| copilotnative | No explicit Allow rules. |
|
| copilotsapphire | No explicit Allow rules. |
|
| imgproxy | No explicit Allow rules. |
|
| semrushbot-swa | No explicit Allow rules. |
|
| chatgpt-user | No explicit Allow rules. |
|
| oai-searchbot | No explicit Allow rules. |
|
| timpibot | No explicit Allow rules. |
|
| thinkbot | No explicit Allow rules. |
|
Blocked and Allowed Paths
| Blocked paths |
|
|---|---|
| Allowed paths |
|
| Crawl-delay | No Crawl-delay directive detected. |
Sitemaps Detected
No Sitemap directives found in robots.txt.
AI Crawler Policy
At least one AI crawler (such as GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot or Google-Extended) appears to be blocked by robots.txt.
Issues Found
- At least one user-agent has Disallow: / which blocks the entire site.
- robots.txt does not reference any sitemap URLs.
- robots.txt contains a very large number of Disallow rules; consider simplifying patterns.
- Wildcard (*) patterns should typically be used with explicit paths, e.g. /path/*.
Recommendations
- Add a Sitemap directive in robots.txt pointing to your primary XML sitemap.
- Avoid blocking the entire site (Disallow: /); restrict only sensitive or low-value paths instead.
- Consolidate overlapping Disallow rules into simpler patterns to keep robots.txt maintainable.
- Review your AI crawler policy for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended to ensure it matches your content strategy.
- Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.