Robots.txt Tester
Inspect how xhaccess.com controls crawler access, blocked paths, sitemap references and AI crawler rules.
Preview
Score: 76
- At least one user-agent has Disallow: / which blocks the entire site.
- robots.txt does not reference any sitemap URLs.
- Wildcard (*) patterns should typically be used with explicit paths, e.g. /path/*.
Get your full report + exact fixes
See what’s hurting your SEO and how to fix it step by step.
- Full breakdown
- Actionable fixes
- Prioritized next steps
Robots.txt Status
Robots.txt Status
Present
Score
76
/100
· Needs review
View Full Robots.txt
Robots.txt Content Preview
User-agent: * disallow: *light=* disallow: /movies/*/download/ disallow: /premium/ allow: /users/ disallow: /users/*/friends disallow: /users/*/favorites disallow: /users/profiles/ disallow: /playlists/ disallow: /posts/ disallow: /embed/ allow: /users/creators allow: /user/xHamster/blog/* allow: /user/xHamster/posts/* disallow: /stories/*/print disallow: *embedded_video* disallow: /photos/gallery/*/ allow: /photos/gallery/*/2$ allow: /photos/gallery/*/3$ allow: /photos/gallery/*/4$ allow: /photos/gallery/*/5$ allow: /photos/gallery/*.html disallow: /ab.log disallow: /hot-7 disallow: /hot-30 disallow: /hot-7-pro disallow: /hot-30-pro disallow: /switch disallow: *?max-duration= disallow: *?min-duration= disallow: *&max-duration= disallow: *&min-duration= disallow: *?geo= disallow: */categories/*/best/daily disallow: */categories/*/best/monthly disallow: */categories/*/best/weekly disallow: */categories/*/best/year-20 disallow: */categories/*/most-commented disallow: */categories/*/most-viewed disallow: */categories/*/producer disallow: */categories/*/creator disallow: */channels/*/newest disallow: */channels/*/best disallow: */channels/*/most-commented disallow: */channels/*/most-viewed disallow: */pornstars/*/newest disallow: */pornstars/*/best disallow: */pornstars/*/most-commented disallow: */pornstars/*/most-viewed disallow: */pornstars/*/exclusive disallow: */creators/*/newest disallow: */creators/*/best disallow: */creators/*/most-commented disallow: */creators/*/most-viewed disallow: */creators/*/exclusive disallow: */creators/*/full-length disallow: */celebrities/*/newest disallow: */celebrities/*/best disallow: */celebrities/*/most-commented disallow: */celebrities/*/most-viewed disallow: */tags/*/newest disallow: */tags/*/best disallow: */tags/*/most-commented disallow: */tags/*/most-viewed disallow: */tags/*/producer disallow: */tags/*/creator allow: */categories/*/best allow: */pornstars/all/*/best allow: */creators/all/*/best allow: */celebrities/all/*/best disallow: */categories/*/best*?max-duration= disallow: */categories/*/best*?min-duration= disallow: */categories/*/best*&max-duration= disallow: */categories/*/best*&min-duration= disallow: /exit.php disallow: */*/*/vr?*dimension= disallow: */*/*/vr?*angle= disallow: */categories/*fps= disallow: */tags/*fps= disallow: */channels/*fps= disallow: */pornstars/*fps= disallow: */creators/*fps= disallow: */celebrities/*fps= disallow: *=producer disallow: *=creator disallow: /signup? disallow: /api/ disallow: */search/*cats= disallow: */search/*correction= disallow: */search/*date= disallow: */search/*length= disallow: */search/*orientations= disallow: */search/*prod= disallow: */search/*revert= disallow: */search/*hidewatched= disallow: */search/*sort=date disallow: */search/*sort=longest disallow: */search/*sort=views disallow: */search/*angle= disallow: */search/*dimension= # Disallow all pagination Disallow: */search/*page= # Allow search pages from 2 to 10 Allow: */search/*?page=2$ Allow: */search/*?page=3$ Allow: */search/*?page=4$ Allow: */search/*?page=5$ Allow: */search/*?page=6$ Allow: */search/*?page=7$ Allow: */search/*?page=8$ Allow: */search/*?page=9$ Allow: */search/*?page=10$ # Allow best search pages from 2 to 10 Allow: */search/*?sort=best&page=2$ Allow: */search/*?sort=best&page=3$ Allow: */search/*?sort=best&page=4$ Allow: */search/*?sort=best&page=5$ Allow: */search/*?sort=best&page=6$ Allow: */search/*?sort=best&page=7$ Allow: */search/*?sort=best&page=8$ Allow: */search/*?sort=best&page=9$ Allow: */search/*?sort=best&page=10$ disallow: *?skip= disallow: *&skip= disallow: /webcams/ disallow: /tracking? disallow: *?x_platform_switch=* disallow: /xembed.php disallow: /livevideoamp/ disallow: /fhwidgetamp/ disallow: /info/list-of-processors disallow: /info/dsa-transparency-report-2024 disallow: /info/dsa-transparency-report-2025 disallow: /live User-agent: ia_archiver Disallow: /
User-agent Rules
| User-agent(s) | Allowed paths | Disallowed paths |
|---|---|---|
| * |
|
|
| ia_archiver | No explicit Allow rules. |
|
Blocked and Allowed Paths
| Blocked paths |
|
|---|---|
| Allowed paths |
|
| Crawl-delay | No Crawl-delay directive detected. |
Sitemaps Detected
No Sitemap directives found in robots.txt.
AI Crawler Policy
No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).
Issues Found
- At least one user-agent has Disallow: / which blocks the entire site.
- robots.txt does not reference any sitemap URLs.
- Wildcard (*) patterns should typically be used with explicit paths, e.g. /path/*.
Recommendations
- Add a Sitemap directive in robots.txt pointing to your primary XML sitemap.
- Avoid blocking the entire site (Disallow: /); restrict only sensitive or low-value paths instead.
- Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
- Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.