Robots.txt Tester

Inspect how tseikurutimes.com controls crawler access, blocked paths, sitemap references and AI crawler rules.

Preview

Score: 92

Sitemap URL appears to be unreachable: https://tseikurutimes.com/sitemap_index.xml

Get your full report + exact fixes

See what’s hurting your SEO and how to fix it step by step.

Full breakdown
Actionable fixes
Prioritized next steps

Robots.txt Status Present Score 92 /100 · Strong

Domain tseikurutimes.com

robots.txt URL https://tseikurutimes.com/robots.txt

Last analyzed June 5, 2026

View Full Robots.txt

Open robots.txt in a new tab

Download robots.txt

Robots.txt Content Preview

# Tseikuru Times - Robots.txt Configuration
# Allows search engine crawlers to index the website

# Allow all crawlers
User-agent: *
Allow: /

# Disallow admin pages
Disallow: /admin
Disallow: /admin/login
Disallow: /debug

# Allow Google and Bing to crawl all other pages
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Explicitly allow other common crawlers (optional — `User-agent: *` already covers these)
User-agent: DuckDuckBot
Allow: /

User-agent: Slurp
Allow: /

User-agent: Yandex
Allow: /

User-agent: Baiduspider
Allow: /

User-agent: Sogou
Allow: /

User-agent: Exabot
Allow: /

User-agent: Facebot
Allow: /

User-agent: Applebot
Allow: /

User-agent: Twitterbot
Allow: /

User-agent: Firefox
Allow: /

# Crawl delay removed to allow all crawlers full access
# (If you need to limit crawl rate, configure via specific crawler tools)

# Sitemap locations (custom domain)
Sitemap: https://tseikurutimes.com/sitemap_index.xml
Sitemap: https://tseikurutimes.com/sitemap.xml

User-agent Rules

User-agent(s)	Allowed paths	Disallowed paths
*	/	/admin /admin/login /debug
googlebot	/	No explicit Disallow rules.
bingbot	/	No explicit Disallow rules.
duckduckbot	/	No explicit Disallow rules.
slurp	/	No explicit Disallow rules.
yandex	/	No explicit Disallow rules.
baiduspider	/	No explicit Disallow rules.
sogou	/	No explicit Disallow rules.
exabot	/	No explicit Disallow rules.
facebot	/	No explicit Disallow rules.
applebot	/	No explicit Disallow rules.
twitterbot	/	No explicit Disallow rules.
firefox	/	No explicit Disallow rules.

Blocked and Allowed Paths

Blocked paths	/admin /admin/login /debug
Allowed paths	/
Crawl-delay	No Crawl-delay directive detected.

Sitemaps Detected

AI Crawler Policy

No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).

Issues Found

Sitemap URL appears to be unreachable: https://tseikurutimes.com/sitemap_index.xml

Recommendations

Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.