Meet our crawlers
Bing operates five main crawlers today:
|Crawler||Role of this Particular Crawler||Example of User Agent String|
|Bingbot||Bingbot is our standard crawler and handles most of our crawling needs each day. Bingbot uses a couple of different user agent strings which include several mobile variants with which we crawl the mobile web (see here for details on the latter).||
|MSNBot||MSNBot used to be our standard crawler before the advent of Bingbot and still handles some of our crawling duties.||
|MSNBot-Media||MSNBot-Media is our crawler for images and video.||
|AdIdxBot||AdIdxBot is the crawler used by Bing Ads. AdIdxBot is responsible for crawling ads and following through to websites from those ads for quality control purposes. Same as Bingbot, AdIdxBot has both desktop and mobile variants.||
|BingPreview||BingPreview is used to generate page snapshots. You can find more details about Bing Preview here. Note that BingPreview also has "desktop" and "mobile" variants.||
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
The above crawlers will identify themselves with the above names in the user agent string. However, user agent strings are easy to spoof, so not every request using these user agent names inside of their user agent string may be coming from a real Bing crawler. As a general rule, Bing does not share the IP addresses from which we crawl the web, but you can always use the Verify Bingbot tool to check whether a crawler actually belongs to Bing.
Controlling Crawl and Crawl Rates
To control how our crawlers interact with your website, you have two options:
- Robots.txt files can be configured to tell our crawlers how to interact with your website
- Through Bing Webmaster tools you can control crawl rates by the hour using the Crawl Control tool.
Should you notice crawl issues with Bingbot or any of our other crawlers, please follow the steps outlined in How to Report and Issue With Bingbot.