Robots.txt Tester

Inspect how facebook.com controls crawler access, blocked paths, sitemap references and AI crawler rules.

Preview

Score: 84

  • At least one user-agent has Disallow: / which blocks the entire site.
  • robots.txt does not reference any sitemap URLs.

Get your full report + exact fixes

See what’s hurting your SEO and how to fix it step by step.

  • Full breakdown
  • Actionable fixes
  • Prioritized next steps

No spam. One email with your report and next steps.

Robots.txt Status Present Score 84 /100 · Strong
Domain facebook.com
robots.txt URL https://facebook.com/robots.txt
Last analyzed June 18, 2026

View Full Robots.txt

Open robots.txt in a new tab

Download robots.txt

Robots.txt Content Preview

# Notice: Collection of data on Facebook through automated means is
# prohibited unless you have express written permission from Facebook
# and may only be conducted for the limited purpose contained in said
# permission.
# All authorized user-agents listed on this page must comply with Meta’s
# Automated Data Collection Terms available at:
# https://www.facebook.com/legal/automated_data_collection_terms
User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Brightbot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: uptimerobot
Disallow: /

User-agent: viberbot
Disallow: /

User-agent: YaK
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Yeti
Disallow: /

User-agent: Applebot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Bingbot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Discordbot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: DuckDuckBot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: facebookexternalhit
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Googlebot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Googlebot-Image
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /map_tile.php
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /static_map.php
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Googlebot-Video
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Googlebot-News
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Google-InspectionTool
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: LinkedInBot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: msnbot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Pinterestbot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Screaming Frog SEO Spider
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: seznambot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Slurp
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: teoma
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: TelegramBot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Twitterbot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: *
Disallow: /

User-agent Rules

User-agent(s) Allowed paths Disallowed paths
amazonbot No explicit Allow rules.
  • /
applebot-extended No explicit Allow rules.
  • /
brightbot No explicit Allow rules.
  • /
claudebot No explicit Allow rules.
  • /
google-extended No explicit Allow rules.
  • /
gptbot No explicit Allow rules.
  • /
perplexitybot No explicit Allow rules.
  • /
petalbot No explicit Allow rules.
  • /
scrapy No explicit Allow rules.
  • /
uptimerobot No explicit Allow rules.
  • /
viberbot No explicit Allow rules.
  • /
yak No explicit Allow rules.
  • /
yandex No explicit Allow rules.
  • /
yeti No explicit Allow rules.
  • /
applebot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
bingbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
discordbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
duckduckbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
facebookexternalhit
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
googlebot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
googlebot-image
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /map_tile.php
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /static_map.php
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
googlebot-video
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
googlebot-news
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
google-inspectiontool
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
linkedinbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
msnbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
pinterestbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
screaming frog seo spider
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
seznambot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
slurp
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
teoma
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
telegrambot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
twitterbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
* No explicit Allow rules.
  • /

Blocked and Allowed Paths

Blocked paths
  • /
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
  • /map_tile.php
  • /static_map.php
Allowed paths
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
Crawl-delay No Crawl-delay directive detected.

Sitemaps Detected

No Sitemap directives found in robots.txt.

AI Crawler Policy

At least one AI crawler (such as GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot or Google-Extended) appears to be blocked by robots.txt.

Issues Found

  • At least one user-agent has Disallow: / which blocks the entire site.
  • robots.txt does not reference any sitemap URLs.

Recommendations

  • Add a Sitemap directive in robots.txt pointing to your primary XML sitemap.
  • Avoid blocking the entire site (Disallow: /); restrict only sensitive or low-value paths instead.
  • Review your AI crawler policy for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended to ensure it matches your content strategy.
  • Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.

Analyze this site with other tools

Want a website that actually generates leads?

Start a conversion-focused website project with a team that builds fast, SEO-optimized sites for real businesses.

Start project