Robots.txt Tester

Inspect how 365education.org controls crawler access, blocked paths, sitemap references and AI crawler rules.

Preview

Score: 84

Sitemap URL appears to be unreachable: https://www.365education.org/news-sitemap.xml
Blocking CSS or JS may prevent search engines from rendering pages correctly.

Get your full report + exact fixes

See what’s hurting your SEO and how to fix it step by step.

Full breakdown
Actionable fixes
Prioritized next steps

Robots.txt Status Present Score 84 /100 · Strong

Domain 365education.org

robots.txt URL https://365education.org/robots.txt

Last analyzed June 19, 2026

View Full Robots.txt

Open robots.txt in a new tab

Download robots.txt

Robots.txt Content Preview

# ============================================================
# robots.txt for https://www.365education.org
# Generated: 2026-06-16
# Purpose: Maximum crawlability, indexing & SEO performance
# ============================================================


# ============================================================
# SEARCH ENGINES â Full Access
# ============================================================

User-agent: Googlebot
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-Video
Allow: /

User-agent: Googlebot-News
Allow: /

User-agent: AdsBot-Google
Allow: /

User-agent: Bingbot
Allow: /

User-agent: AdIdxBot
Allow: /

User-agent: BingPreview
Allow: /

User-agent: Slurp
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Baiduspider
Allow: /

User-agent: YandexBot
Allow: /

User-agent: YandexImages
Allow: /

User-agent: YandexVideo
Allow: /

User-agent: YandexNews
Allow: /

User-agent: Sogou web spider
Allow: /

User-agent: Sogou inst spider
Allow: /

User-agent: Exabot
Allow: /

User-agent: ia_archiver
Allow: /

User-agent: archive.org_bot
Allow: /

User-agent: Applebot
Allow: /

User-agent: NaverBot
Allow: /

User-agent: Seznam robot
Allow: /

User-agent: BLEXBot
Allow: /

User-agent: MojeekBot
Allow: /

User-agent: Qwantify
Allow: /


# ============================================================
# SOCIAL MEDIA PLATFORMS â Full Access (link previews & sharing)
# ============================================================

User-agent: facebookexternalhit
Allow: /

User-agent: Facebot
Allow: /

User-agent: Twitterbot
Allow: /

User-agent: LinkedInBot
Allow: /

User-agent: WhatsApp
Allow: /

User-agent: Pinterestbot
Allow: /

User-agent: Slackbot
Allow: /

User-agent: Slackbot-LinkExpanding
Allow: /

User-agent: TelegramBot
Allow: /

User-agent: Discordbot
Allow: /

User-agent: SkypeUriPreview
Allow: /

User-agent: Snapchat
Allow: /

User-agent: redditbot
Allow: /

User-agent: TikTok
Allow: /

User-agent: vkShare
Allow: /

User-agent: Tumblr
Allow: /

User-agent: Embedly
Allow: /


# ============================================================
# AI PLATFORMS & LARGE LANGUAGE MODELS â Full Access
# ============================================================

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: YouBot
Allow: /

User-agent: Cohere-ai
Allow: /

User-agent: cohere-training-data-crawler
Allow: /

User-agent: Omgili
Allow: /

User-agent: omgilibot
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: Meta-ExternalFetcher
Allow: /

User-agent: Bytespider
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: GoogleExtended
Allow: /

User-agent: DuplexWeb-Google
Allow: /


# ============================================================
# SEO TOOLS & AUDITING BOTS â Full Access
# ============================================================

User-agent: AhrefsBot
Allow: /

User-agent: SemrushBot
Allow: /

User-agent: SemrushBot-SA
Allow: /

User-agent: MJ12bot
Allow: /

User-agent: DotBot
Allow: /

User-agent: rogerbot
Allow: /

User-agent: Screaming Frog SEO Spider
Allow: /

User-agent: seobilitybot
Allow: /

User-agent: DataForSeoBot
Allow: /


# ============================================================
# CATCH-ALL â Allow all other crawlers by default
# ============================================================

User-agent: *
Allow: /

# Disallow only low-value, duplicate, or system paths
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /wp-includes/
Disallow: /wp-json/
Disallow: /xmlrpc.php
Disallow: /feed/
Disallow: /*/feed/
Disallow: /*/trackback/
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /?s=
Disallow: /search/
Disallow: /tag/
Disallow: /?p=
Disallow: /page/
Disallow: /checkout/
Disallow: /cart/
Disallow: /my-account/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /*.php$


# ============================================================
# CRAWL DELAY (optional: reduce server load from heavy bots)
# ============================================================

User-agent: AhrefsBot
Crawl-delay: 2

User-agent: SemrushBot
Crawl-delay: 2

User-agent: MJ12bot
Crawl-delay: 3


# ============================================================
# SITEMAPS â Tell all crawlers where to find content
# ============================================================

Sitemap: https://www.365education.org/sitemap.xml
Sitemap: https://www.365education.org/sitemap_index.xml
Sitemap: https://www.365education.org/post-sitemap.xml
Sitemap: https://www.365education.org/page-sitemap.xml
Sitemap: https://www.365education.org/category-sitemap.xml
Sitemap: https://www.365education.org/news-sitemap.xml

User-agent Rules

User-agent(s)	Allowed paths	Disallowed paths
googlebot	/	No explicit Disallow rules.
googlebot-image	/	No explicit Disallow rules.
googlebot-video	/	No explicit Disallow rules.
googlebot-news	/	No explicit Disallow rules.
adsbot-google	/	No explicit Disallow rules.
bingbot	/	No explicit Disallow rules.
adidxbot	/	No explicit Disallow rules.
bingpreview	/	No explicit Disallow rules.
slurp	/	No explicit Disallow rules.
duckduckbot	/	No explicit Disallow rules.
baiduspider	/	No explicit Disallow rules.
yandexbot	/	No explicit Disallow rules.
yandeximages	/	No explicit Disallow rules.
yandexvideo	/	No explicit Disallow rules.
yandexnews	/	No explicit Disallow rules.
sogou web spider	/	No explicit Disallow rules.
sogou inst spider	/	No explicit Disallow rules.
exabot	/	No explicit Disallow rules.
ia_archiver	/	No explicit Disallow rules.
archive.org_bot	/	No explicit Disallow rules.
applebot	/	No explicit Disallow rules.
naverbot	/	No explicit Disallow rules.
seznam robot	/	No explicit Disallow rules.
blexbot	/	No explicit Disallow rules.
mojeekbot	/	No explicit Disallow rules.
qwantify	/	No explicit Disallow rules.
facebookexternalhit	/	No explicit Disallow rules.
facebot	/	No explicit Disallow rules.
twitterbot	/	No explicit Disallow rules.
linkedinbot	/	No explicit Disallow rules.
whatsapp	/	No explicit Disallow rules.
pinterestbot	/	No explicit Disallow rules.
slackbot	/	No explicit Disallow rules.
slackbot-linkexpanding	/	No explicit Disallow rules.
telegrambot	/	No explicit Disallow rules.
discordbot	/	No explicit Disallow rules.
skypeuripreview	/	No explicit Disallow rules.
snapchat	/	No explicit Disallow rules.
redditbot	/	No explicit Disallow rules.
tiktok	/	No explicit Disallow rules.
vkshare	/	No explicit Disallow rules.
tumblr	/	No explicit Disallow rules.
embedly	/	No explicit Disallow rules.
gptbot	/	No explicit Disallow rules.
chatgpt-user	/	No explicit Disallow rules.
oai-searchbot	/	No explicit Disallow rules.
ccbot	/	No explicit Disallow rules.
anthropic-ai	/	No explicit Disallow rules.
claudebot	/	No explicit Disallow rules.
claude-web	/	No explicit Disallow rules.
perplexitybot	/	No explicit Disallow rules.
youbot	/	No explicit Disallow rules.
cohere-ai	/	No explicit Disallow rules.
cohere-training-data-crawler	/	No explicit Disallow rules.
omgili	/	No explicit Disallow rules.
omgilibot	/	No explicit Disallow rules.
meta-externalagent	/	No explicit Disallow rules.
meta-externalfetcher	/	No explicit Disallow rules.
bytespider	/	No explicit Disallow rules.
applebot-extended	/	No explicit Disallow rules.
googleextended	/	No explicit Disallow rules.
duplexweb-google	/	No explicit Disallow rules.
ahrefsbot	/	No explicit Disallow rules.
semrushbot	/	No explicit Disallow rules.
semrushbot-sa	/	No explicit Disallow rules.
mj12bot	/	No explicit Disallow rules.
dotbot	/	No explicit Disallow rules.
rogerbot	/	No explicit Disallow rules.
screaming frog seo spider	/	No explicit Disallow rules.
seobilitybot	/	No explicit Disallow rules.
dataforseobot	/	No explicit Disallow rules.
*	/	/wp-admin/ /wp-login.php /wp-includes/ /wp-json/ /xmlrpc.php /feed/ //feed/ //trackback/ /cgi-bin/ /tmp/ /?s= /search/ /tag/ /?p= /page/ /checkout/ /cart/ /my-account/ /wp-content/plugins/ /wp-content/themes/ /*.php$
ahrefsbot	No explicit Allow rules.	No explicit Disallow rules.
semrushbot	No explicit Allow rules.	No explicit Disallow rules.
mj12bot	No explicit Allow rules.	No explicit Disallow rules.

Blocked and Allowed Paths

Blocked paths	/wp-admin/ /wp-login.php /wp-includes/ /wp-json/ /xmlrpc.php /feed/ //feed/ //trackback/ /cgi-bin/ /tmp/ /?s= /search/ /tag/ /?p= /page/ /checkout/ /cart/ /my-account/ /wp-content/plugins/ /wp-content/themes/ /*.php$
Allowed paths	/
Crawl-delay	3.0 seconds

Sitemaps Detected

AI Crawler Policy

No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).

Issues Found

Sitemap URL appears to be unreachable: https://www.365education.org/news-sitemap.xml
Blocking CSS or JS may prevent search engines from rendering pages correctly.

Recommendations

Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.