Robots.txt Tester

Inspect how gist.github.com controls crawler access, blocked paths, sitemap references and AI crawler rules.

Preview

Score: 84

robots.txt does not reference any sitemap URLs.
Wildcard (*) patterns should typically be used with explicit paths, e.g. /path/*.

Get your full report + exact fixes

See what’s hurting your SEO and how to fix it step by step.

Full breakdown
Actionable fixes
Prioritized next steps

Robots.txt Status Present Score 84 /100 · Strong

Domain gist.github.com

robots.txt URL https://gist.github.com/robots.txt

Last analyzed June 9, 2026

View Full Robots.txt

Open robots.txt in a new tab

Download robots.txt

Robots.txt Content Preview

# If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots
# We also provide an extensive API: https://docs.github.com
User-agent: bingbot
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /account-login
Disallow: */tarball/
Disallow: */zipball/
Disallow: /Explodingstuff/
Disallow: /copilot/
Disallow: /copilot/c/

User-agent: adidxbot
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /account-login
Disallow: */tarball/
Disallow: */zipball/
Disallow: /Explodingstuff/
Disallow: /copilot/
Disallow: /copilot/c/

User-agent: BingPreview
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /account-login
Disallow: */tarball/
Disallow: */zipball/
Disallow: /Explodingstuff/
Disallow: /copilot/
Disallow: /copilot/c/

User-agent: baidu
crawl-delay: 1


User-agent: *

Disallow: /*/*/pulse
Disallow: /*/*/projects
Disallow: /*/*/forks
Disallow: /*/*/issues/new
Disallow: /*/*/milestones/new
Disallow: /*/*/issues/search
Disallow: /*/*/commits/
Disallow: /*/*/branches
Disallow: /*/*/contributors
Disallow: /*/*/tags
Disallow: /*/*/stargazers
Disallow: /*/*/watchers
Disallow: /*/*/network
Disallow: /*/*/graphs
Disallow: /*/*/compare

Disallow: /*/tree/
Disallow: /gist/
Disallow: /*/download
Disallow: /*/revisions
Disallow: /*/commits/*?author
Disallow: /*/commits/*?path
Disallow: /*/comments
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/raw/
Disallow: /*/cache/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /search/advanced
Disallow: /search$
Disallow: /*q=
Disallow: /*.atom$

Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: */tarball/
Disallow: */zipball/

Disallow: /*source=*
Disallow: /*ref_cta=*
Disallow: /*plan=*
Disallow: /*return_to=*
Disallow: /*ref_loc=*
Disallow: /*setup_organization=*
Disallow: /*source_repo=*
Disallow: /*ref_page=*
Disallow: /*source=*
Disallow: /*referrer=*
Disallow: /*report=*
Disallow: /*author=*
Disallow: /*since=*
Disallow: /*until=*
Disallow: /*commits?author=*
Disallow: /*report-abuse?report=*
Disallow: /*tab=*
Allow: /*?tab=achievements&achievement=*

Disallow: /account-login
Disallow: /Explodingstuff/

Disallow: /copilot/
Disallow: /copilot/c/

User-agent Rules

User-agent(s)	Allowed paths	Disallowed paths
bingbot	No explicit Allow rules.	/ekansa/open-context-data /ekansa/opencontext-* /account-login /tarball/ /zipball/ /explodingstuff/ /copilot/ /copilot/c/
adidxbot	No explicit Allow rules.	/ekansa/open-context-data /ekansa/opencontext-* /account-login /tarball/ /zipball/ /explodingstuff/ /copilot/ /copilot/c/
bingpreview	No explicit Allow rules.	/ekansa/open-context-data /ekansa/opencontext-* /account-login /tarball/ /zipball/ /explodingstuff/ /copilot/ /copilot/c/
baidu	No explicit Allow rules.	No explicit Disallow rules.
*	/?tab=achievements&achievement=	///pulse ///projects ///forks ///issues/new ///milestones/new ///issues/search ///commits/ ///branches ///contributors ///tags ///stargazers ///watchers ///network ///graphs ///compare //tree/ /gist/ //download //revisions //commits/?author //commits/?path //comments //archive/ //blame/ //raw/ //cache/ /.git/ /.git/ /.git$ /search/advanced /search$ /q= /.atom$ /ekansa/open-context-data /ekansa/opencontext-* /tarball/ /zipball/ /source= /ref_cta= /plan= /return_to= /ref_loc= /setup_organization= /source_repo= /ref_page= /source= /referrer= /report= /author= /since= /until= /commits?author= /report-abuse?report= /tab= /account-login /explodingstuff/ /copilot/ /copilot/c/

Blocked and Allowed Paths

Blocked paths	/ekansa/open-context-data /ekansa/opencontext-* /account-login /tarball/ /zipball/ /explodingstuff/ /copilot/ /copilot/c/ ///pulse ///projects ///forks ///issues/new ///milestones/new ///issues/search ///commits/ ///branches ///contributors ///tags ///stargazers ///watchers ///network ///graphs ///compare //tree/ /gist/ //download //revisions //commits/?author //commits/?path //comments //archive/ //blame/ //raw/ //cache/ /.git/ /.git/ /.git$ /search/advanced /search$ /q= /.atom$ /source= /ref_cta= /plan= /return_to= /ref_loc= /setup_organization= /source_repo= /ref_page= /referrer= /report= /author= /since= /until= /commits?author= /report-abuse?report= /tab=
Allowed paths	/?tab=achievements&achievement=
Crawl-delay	1.0 seconds

Sitemaps Detected

No Sitemap directives found in robots.txt.

AI Crawler Policy

No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).

Issues Found

robots.txt does not reference any sitemap URLs.
Wildcard (*) patterns should typically be used with explicit paths, e.g. /path/*.

Recommendations

Add a Sitemap directive in robots.txt pointing to your primary XML sitemap.
Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.