Robots.txt Tester
Inspect how aihrbd.com controls crawler access, blocked paths, sitemap references and AI crawler rules.
Preview
Score: 100
- Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
- Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.
Get your full report + exact fixes
See what’s hurting your SEO and how to fix it step by step.
- Full breakdown
- Actionable fixes
- Prioritized next steps
Robots.txt Status
Robots.txt Status
Present
Score
100
/100
· Strong
View Full Robots.txt
Robots.txt Content Preview
# AIHR BD robots.txt # Public marketing pages are crawlable # Private HR/payroll app routes are blocked from indexing Sitemap: https://www.aihrbd.com/sitemap.xml User-agent: * # Public website pages Allow: / Allow: /about Allow: /blog Allow: /contact Allow: /docs Allow: /features Allow: /features/attendance Allow: /features/payroll Allow: /features/leave Allow: /features/devices Allow: /features/performance Allow: /features/employees Allow: /pricing Allow: /customers Allow: /industries Allow: /careers Allow: /partners Allow: /security Allow: /help Allow: /refund Allow: /changelog Allow: /signup Allow: /privacy Allow: /terms # Block authentication & internal app areas Disallow: /login Disallow: /*/login Disallow: /forgot-password Disallow: /reset-password Disallow: /unsubscribe Disallow: /app Disallow: /quick-books-callback # Block all dashboard / workspace routes Disallow: /employees Disallow: /attendance Disallow: /leave Disallow: /payroll Disallow: /reports Disallow: /notices Disallow: /notifications Disallow: /devices Disallow: /directory Disallow: /company Disallow: /subscription Disallow: /settings Disallow: /audit-logs Disallow: /holidays Disallow: /performance Disallow: /performance-reviews Disallow: /shifts Disallow: /overtime Disallow: /breaks Disallow: /meals Disallow: /recruitment Disallow: /loans Disallow: /bonuses Disallow: /geofences Disallow: /assets Disallow: /vehicles Disallow: /vehicle-documents Disallow: /drivers Disallow: /driver-documents Disallow: /driver-profile Disallow: /driver-profile-requests Disallow: /driver-portal Disallow: /vehicle-requisitions Disallow: /conveyance-bills Disallow: /trip-logs Disallow: /my-trip-logs Disallow: /trip-receipt-reviews Disallow: /permissions Disallow: /letters Disallow: /departments Disallow: /branches Disallow: /designations Disallow: /organization Disallow: /goals Disallow: /training Disallow: /knowledge-base Disallow: /compliance Disallow: /integrations Disallow: /salary-templates Disallow: /documents Disallow: /my-portal Disallow: /my-payslips Disallow: /my-documents Disallow: /my-profile Disallow: /companies # Admin areas Disallow: /super-admin Disallow: /super-admin/ # Allow assets for rendering Allow: /assets/ Allow: /images/ Allow: /static/ Allow: /public/ # AI crawlers User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / # Search engine crawlers User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: DuckDuckBot Allow: / User-agent: YandexBot Allow: / User-agent: CCBot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: cohere-ai Allow: /
User-agent Rules
| User-agent(s) | Allowed paths | Disallowed paths |
|---|---|---|
| * |
|
|
| gptbot |
|
No explicit Disallow rules. |
| chatgpt-user |
|
No explicit Disallow rules. |
| oai-searchbot |
|
No explicit Disallow rules. |
| claudebot |
|
No explicit Disallow rules. |
| claude-web |
|
No explicit Disallow rules. |
| anthropic-ai |
|
No explicit Disallow rules. |
| perplexitybot |
|
No explicit Disallow rules. |
| perplexity-user |
|
No explicit Disallow rules. |
| google-extended |
|
No explicit Disallow rules. |
| googlebot |
|
No explicit Disallow rules. |
| bingbot |
|
No explicit Disallow rules. |
| applebot |
|
No explicit Disallow rules. |
| applebot-extended |
|
No explicit Disallow rules. |
| duckduckbot |
|
No explicit Disallow rules. |
| yandexbot |
|
No explicit Disallow rules. |
| ccbot |
|
No explicit Disallow rules. |
| meta-externalagent |
|
No explicit Disallow rules. |
| cohere-ai |
|
No explicit Disallow rules. |
Blocked and Allowed Paths
| Blocked paths |
|
|---|---|
| Allowed paths |
|
| Crawl-delay | No Crawl-delay directive detected. |
Sitemaps Detected
AI Crawler Policy
No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).
Recommendations
- Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
- Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.