Robots.txt Tester

Inspect how thailabhub.com controls crawler access, blocked paths, sitemap references and AI crawler rules.

Preview

Score: 100

  • Review your AI crawler policy for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended to ensure it matches your content strategy.
  • Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.

Get your full report + exact fixes

See what’s hurting your SEO and how to fix it step by step.

  • Full breakdown
  • Actionable fixes
  • Prioritized next steps

No spam. One email with your report and next steps.

Robots.txt Status Present Score 100 /100 · Strong
Domain thailabhub.com
robots.txt URL https://thailabhub.com/robots.txt
Last analyzed June 2, 2026

View Full Robots.txt

Open robots.txt in a new tab

Download robots.txt

Robots.txt Content Preview

User-Agent: *
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: GPTBot
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: ChatGPT-User
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: OAI-SearchBot
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: anthropic-ai
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: ClaudeBot
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: PerplexityBot
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: CCBot
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: Google-Extended
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: FacebookBot
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: Meta-ExternalAgent
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

User-Agent: Applebot-Extended
Allow: /
Disallow: /api/
Disallow: /th/admin/
Disallow: /th/dashboard/
Disallow: /th/factory/
Disallow: /th/portal/
Disallow: /th/desktop/
Disallow: /th/payment/
Disallow: /th/rfq/
Disallow: /th/store/
Disallow: /en/admin/
Disallow: /en/dashboard/
Disallow: /en/factory/
Disallow: /en/portal/
Disallow: /en/desktop/
Disallow: /en/payment/
Disallow: /en/rfq/
Disallow: /en/store/

Sitemap: https://thailabhub.com/sitemap.xml
Sitemap: https://thailabhub.com/llms.txt

User-agent Rules

User-agent(s) Allowed paths Disallowed paths
*
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
gptbot
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
chatgpt-user
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
oai-searchbot
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
anthropic-ai
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
claudebot
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
perplexitybot
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
ccbot
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
google-extended
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
facebookbot
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
meta-externalagent
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
applebot-extended
  • /
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/

Blocked and Allowed Paths

Blocked paths
  • /api/
  • /th/admin/
  • /th/dashboard/
  • /th/factory/
  • /th/portal/
  • /th/desktop/
  • /th/payment/
  • /th/rfq/
  • /th/store/
  • /en/admin/
  • /en/dashboard/
  • /en/factory/
  • /en/portal/
  • /en/desktop/
  • /en/payment/
  • /en/rfq/
  • /en/store/
Allowed paths
  • /
Crawl-delay No Crawl-delay directive detected.

Sitemaps Detected

AI Crawler Policy

At least one AI crawler (such as GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot or Google-Extended) appears to be blocked by robots.txt.

Recommendations

  • Review your AI crawler policy for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended to ensure it matches your content strategy.
  • Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.

Analyze this site with other tools

Want a website that actually generates leads?

Start a conversion-focused website project with a team that builds fast, SEO-optimized sites for real businesses.

Start project