Meet our crawlers
Bing operates five main crawlers today:
|Crawler||Role of this Particular Crawler||Example of User Agent String|
|Bingbot||Bingbot is our standard crawler and handles most of our crawling needs each day. Bingbot uses a couple of different user agent strings.||
We are committing to regularly update our web page rendering engine to the most recent stable version of Microsoft Edge. Thus, "W.X.Y.Z" will be substituted with the latest Microsoft Edge version we're using, for example “80.0.345.0".
|AdIdxBot||AdIdxBot is the crawler used by Bing Ads. AdIdxBot is responsible for crawling ads and following through to websites from those ads for quality control purposes. Same as Bingbot, AdIdxBot has both desktop and mobile variants.||
|BingPreview||BingPreview is used to generate page snapshots. You can find more details about Bing Preview here. Note that BingPreview also has "desktop" and "mobile" variants.||
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
The above crawlers will identify themselves with the above names in the user agent string. However, user agent strings are easy to spoof, so not every request using these user agent names inside of their user agent string may be coming from a real Bing crawler. As a general rule, Bing does not share the IP addresses from which we crawl the web, but you can always use the Verify Bingbot tool to check whether a crawler actually belongs to Bing.
Controlling Crawl and Crawl Rates
To control how our crawlers interact with your website, you have two options:
- Robots.txt files can be configured to tell our crawlers how to interact with your website
- Through Bing Webmaster tools you can control crawl rates by the hour using the Crawl Control tool.
Should you notice crawl issues with Bingbot or any of our other crawlers, please follow the steps outlined in How to Report and Issue With Bingbot.