Is there a dedicated Copilot crawler I can allow or block?

No. Copilot has no standalone crawler — it grounds answers in the Bing index built by Bingbot. Access control for Copilot happens through Bingbot in robots.txt plus the nocache and noarchive meta tags Bing introduced for AI answer control in September 2023.

How do I verify that a visitor claiming to be Bingbot is genuine?

Run a reverse DNS lookup on the IP: genuine Bingbot IPs resolve to hostnames ending in search.msn.com, and a forward lookup on that hostname returns the same IP. Microsoft also publishes a downloadable list of Bingbot IP ranges. Never trust the user-agent string alone.

Will blocking GPTBot affect my Copilot visibility?

No. GPTBot is OpenAI's training crawler and has no relationship to Microsoft Copilot's retrieval. Copilot depends on Bingbot. The two decisions are independent — many publishers block GPTBot while keeping Bingbot fully open.

Microsoft Copilot Crawlers Explained: User Agents, robots.txt and Access Control

Last updated: July 5, 2026 · Menra Team

Microsoft Copilot does not operate its own crawler. Its answers ground in the Bing index, so the user agents that matter are Bingbot (Bing's primary crawler), MicrosoftPreview (page-preview generation across Microsoft products), and AdIdxBot (ads landing-page checks). Controlling Copilot's use of your content happens through Bingbot access plus two meta tags — nocache and noarchive — that Bing wired to AI answers in September 2023.

The user agents

Agent	UA string contains	Purpose	Blocking effect
Bingbot	`bingbot/2.0; +http://www.bing.com/bingbot.htm`	Crawl and index for Bing Search	Removes you from Bing and Copilot
MicrosoftPreview	`MicrosoftPreview/2.0`	Link previews in Microsoft surfaces	Degraded previews, links still work
AdIdxBot	`adidxbot/2.0`	Microsoft Advertising landing checks	Ad quality review fails

Bingbot's full UA is desktop-Chrome-shaped: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36. Match on the bingbot token, not the full string, since Chrome version numbers rotate.

robots.txt configurations

The recommended default for any site that wants Copilot citations:

User-agent: bingbot
Allow: /

User-agent: bingbot
Disallow: /account/
Disallow: /cart/

Directives are read per user-agent group, and Bing honors Allow, Disallow, and wildcard patterns. Bing also respects Crawl-delay, unlike Google — but prefer setting crawl rate in Bing Webmaster Tools, where you get scheduling control without slowing indexation of fresh content.

The AI-answer control layer: nocache and noarchive

Bingbot access is binary; the meta tags add gradation. Bing's September 2023 publisher controls work as follows: <meta name="robots" content="nocache"> limits Copilot to your URL, title, and snippet when grounding — you stay linkable but unquotable. noarchive excludes your content from Copilot answers entirely while preserving normal Bing search listings. Audit templates before assuming your current state is intentional: security-hardening presets in older CMS setups added noarchive broadly, silently opting sites out of AI answers years before those answers existed.

Verification: never trust the UA string

Scrapers impersonate Bingbot constantly. Genuine Bingbot traffic verifies by reverse DNS: the IP resolves to a hostname under search.msn.com, and forward-resolving that hostname returns the same IP. Microsoft additionally publishes a JSON list of official Bingbot IP ranges, which you can load into your WAF as an allowlist. Configure the WAF to exempt verified Bingbot from CAPTCHAs, JavaScript challenges, and rate limits — bot-management defaults on Cloudflare, Akamai, and Fastly regularly throttle Bingbot harder than Googlebot, and every challenged fetch is a page Copilot cannot cite.

Safe WAF and CDN posture

Three rules keep access safe without opening the door to abuse. First, allowlist by verified IP range, not user-agent match. Second, log Bingbot fetch volume weekly; a sudden drop is your earliest warning that a firewall rule changed. Third, serve bots the same HTML humans get — Bing treats deceptive cloaking as spam, though serving prerendered equivalent content is fine. For the broader technical stack behind Copilot visibility — rendering, IndexNow, canonical hygiene — see the technical requirements guide, and for how crawler access fits the full pipeline from index to citation, start with our GEO optimization guide. Definitions for every crawler term used here live in the glossary.

Frequently asked questions

Is there a dedicated Copilot crawler I can allow or block?: No. Copilot has no standalone crawler — it grounds answers in the Bing index built by Bingbot. Access control for Copilot happens through Bingbot in robots.txt plus the nocache and noarchive meta tags Bing introduced for AI answer control in September 2023.
How do I verify that a visitor claiming to be Bingbot is genuine?: Run a reverse DNS lookup on the IP: genuine Bingbot IPs resolve to hostnames ending in search.msn.com, and a forward lookup on that hostname returns the same IP. Microsoft also publishes a downloadable list of Bingbot IP ranges. Never trust the user-agent string alone.
Will blocking GPTBot affect my Copilot visibility?: No. GPTBot is OpenAI's training crawler and has no relationship to Microsoft Copilot's retrieval. Copilot depends on Bingbot. The two decisions are independent — many publishers block GPTBot while keeping Bingbot fully open.

Keep exploring

See how AI engines talk about your brand — track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra