What Is Meta-ExternalAgent? Meta's AI Crawler Explained
Meta-ExternalAgent is the web crawler Meta introduced in August 2024 to collect public web content for training its Llama models and improving Meta AI products. It identifies itself with a user-agent string containing meta-externalagent/1.1 and is the primary robots.txt token publishers use to control Meta's AI data collection.
Which crawlers make up the Meta family?
Meta operates several distinct agents, and conflating them is a common robots.txt mistake. Each serves a different function and carries different visibility consequences.
| Agent | Purpose | Respects robots.txt |
|---|---|---|
| Meta-ExternalAgent | AI training and product improvement | Yes |
| Meta-ExternalFetcher | Fetches a URL when a Meta AI user's request needs it | May bypass (user-initiated) |
| FacebookBot | Legacy crawler for speech/AI research | Yes |
| facebookexternalhit | Link previews when URLs are shared on Meta apps | Not designed to |
Blocking facebookexternalhit breaks link previews across Facebook, Instagram, and WhatsApp — a social-distribution cost that has nothing to do with AI training. Keep the decisions separate.
Why does access matter for Meta AI answers?
Meta AI is the assistant layer inside WhatsApp, Instagram, Facebook, and Messenger; Meta reported it passed one billion monthly active users in mid-2025. When Meta AI answers a question with web context, content that Meta's agents can fetch is what gets represented. Sites that blanket-block all Meta agents remove themselves from one of the largest conversational surfaces on the internet, often unintentionally through a CDN bot-management default rather than a policy decision.
How should a GEO team configure it?
Audit your robots.txt and WAF rules for each Meta token individually. A common posture: allow Meta-ExternalFetcher (it serves live user requests, analogous to ChatGPT-User), decide on Meta-ExternalAgent based on your training-data stance, and never block facebookexternalhit. Then verify actual behavior in server logs, because a Cloudflare or Akamai rule can override whatever robots.txt says. Teams that track AI mentions across engines should confirm crawl access before interpreting a Meta AI visibility gap as a content problem — it is frequently an access problem. Menra's citation tracking helps separate the two by showing which engines actually surface your pages.
Frequently asked questions
- Does Meta-ExternalAgent respect robots.txt?
- Yes. Meta documents that Meta-ExternalAgent honors robots.txt directives, and blocking it is the supported way to opt out of Meta AI training crawls. Meta-ExternalFetcher, the user-triggered fetcher, may bypass robots.txt because it acts on a specific user request rather than crawling autonomously.
- Should I block Meta-ExternalAgent?
- Only if you have a deliberate content-licensing position. Meta AI is embedded in WhatsApp, Instagram, Facebook, and Messenger, reaching roughly a billion monthly users, so blocking the crawler trades that answer-surface reach for training-data control.
Keep exploring
See how AI engines talk about your brand — track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra