Bing blogs

This is a place devoted to giving you deeper insight
into the news, trends, people and technology behind Bing.

Webmaster Blog

September
03

Bingbot is coming to town

Back in June, 2010, we published a blog post titled Bing crawler: bingbot on the horizon that announced our plans to retire our venerable web crawler, MSNBot, and replace it with the new. Our plans remain on track, and we want to remind you that this change will occur on October 1st, 2010.

We also want to take this opportunity to help set some expectations for this process, to discuss the name change details, what the change means in terms of control crawler access to your content, and finally cover what is not changing as part of this process.

Name change

As of October 1st, you'll start seeing the following user agent name in your server logs:

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The old HTTP header From field, currently seen as:

From: msnbot(at)microsoft.com

will be renamed to the following:

From: bingbot(at)microsoft.com

On October 1st, you should see the main Bing web crawler name change implemented. Other MSNBot crawlers will be phased out shortly thereafter.

The renamed crawler will continue to perform as it did previously. You should not see any differences stemming from this change.

Bot access and control

It is critically important that you review your server configurations now for crawler access. Webmaster-managed bot access control methodologies, such as robots.txt files, robots <meta> tag directives, and HTTP headers, may need to be adjusted if you normally block all crawlers except for specific, preferred search crawlers.

Note that we'll still honor all applicable robots.txt directives, including those specific to MSNBot. That said, as of October 1st, if we detect separate sets of custom directives for both MSNBot and BingBot in your robots.txt file, the BingBot directives will take precedence.

If your sites are configured for any special handling based on user agent (such as via the .htaccess file), you'll need to update this for BingBot so we can continue to crawl your sites as before.

To keep your content in the Bing index, please ensure that BingBot has the same access to your site as given to MSNBot. Given the recent developments with Microsoft and Yahoo! Search Alliance, where Bing is now also powering Yahoo! search results for web, image, and video content, this issue is all the more important.

Note: If you previously blocked the Bing crawler but allowed the Yahoo! crawler access to your site, you'll now need to allow access to BingBot to continue to be seen in Yahoo! search results.

What's not changing

The IP addresses used by our crawlers will remain the same. The crawl rate will not be affected by the name change. And, of course, the best way to get BingBot to crawl and index more of your site is to use standard search engine optimization (SEO) techniques. For more information, see our series of posts on performing SEO site reviews, starting with Is your site ranking rank? Do a site review - Part 1 (SEM 101).

We also highly recommend registering your sites with the new Bing Webmaster Tools, where you can add or block URLs, submit Sitemaps, check for crawler-detected problems, get tons of information about your site, and much more.

If you have any questions, comments, or suggestions, feel free to post them in our Indexing and Ranking Discussion forum. Later...

-- Rick DeJarnette, Bing Webmaster Center

Comments

  • Thank You for this update

  • This blog post is more detailed than the one published in June, and has cleared some of my doubts.

  • Thanks for this information!!

  • thanks..

  • Good news.

  • just found this I been working on seo for google. I think bing is better. thanks for the info. I trying to get my site submitted to bing now.

    dyuane

  • like this blogs

  • Thanks Rick, for sharing this information about the Bing BOT.

  • How about fixing the site verification problem in webmaster tools

  • awesome

    google is out now !!

  • Good, I hope its also more frequent a visitor than its predecessor!

  • With this new progress, Bing and Yahoo could surely get even with Google. It is really great to have our websites ranked well in this three search engines

  • Looking forward to the BingBot crawling my websites!

  • Cool

  • Cool :)