Bing blogs

This is a place devoted to giving you deeper insight
into the news, trends, people and technology behind Bing.

Webmaster Blog

June
28

Bing crawler: bingbot on the horizon

Since our last post in November, the Bing team has been busy rolling out improvements to the Bing web crawler. As a result of this work, we want to announce in advance our plans to change the name of our crawler (aka user agent).

Out of beta with a new name

On October 1st, 2010, we will drop the beta designation from the Bing crawler and change the name of the crawler to reflect Microsoft's new brand for search. Instead of the old msnbot 2.0b showing up in your server logs, the updated user agent will be:

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The HTTP header From field will also change as shown below:

From: msnbot(at)microsoft.com

will become

From: bingbot(at)microsoft.com

Robots Exclusion Protocol directives precedence

We want webmasters to know that bingbot will still honor robots.txt directives written for msnbot, so no change is required to your robots.txt file(s).

Please note, however, that if we detect separate sets of directives for bingbot and for any of the older versions of Microsoft search bots (such as msnbot) or a set of directives for all crawlers, the directives for bingbot will take precedence. For example, in the following case, Bing will be authorized to crawl on all whole hosts except on the folder /folder1/, despite the more comprehensive blocking directives for other crawlers:

User-agent: bingbot
Disallow: /folder1/

User-agent: msnbot
Disallow: /folder1/
Disallow: /folder2/

User-agent: *
Disallow: /

How to report issues

We always welcome feedback from the webmaster community. Over the past few months, during the recent development phase, we had different issues reported to us. We want to give our heartfelt thanks to the webmasters who contacted us, and offer our sincere apologies for any issues they may have encountered as a result of the crawler traffic.

If you have any questions regarding BingBot traffic and your site, please contact us at bingbot@microsoft.com. This is the most effective way to reach us. We do our best to respond to your inquiries promptly and report back to you on issues brought to our attention. When you do so, please provide us with a snippet of your logs showing the BingBot traffic as well as a list of the sites served from the same infrastructure, if there is more than one. This way, we can more quickly diagnose and resolve the issues in question.

We look forward to seeing your comments and feedback. Thanks!

Additional bot-related postings:

-- Dean Wierman and Fabrice Canel, Bing Crawl and Index teams

Comments

  • The change of name has been long overdue, and many have been complaining that this needed to be done, to reflect the new face of search.

  • Nice info i too made a post about this in my blog

    www.caxid.com/.../a-new-face-of-bing-crawler-bing-bot

  • MSNbot never (and by never I mean NEVER) crawled my website since Live Search became Bing. I've submitted my site's sitemap several times - with no effect! I mean... c'mon... even Yahoo has indexed it! *sigh* It's quite frustrating... :-(

  • @inaimathi,

    Open a thread in the Webmaster Forum section about the issue you're having, and we will look into it and try to resolve it.

  • This seems much better.

  • Its good that Bing also supports rel nofollow, its a much easier way to block some pages. using disallow, its easy to make mistakes and drop pages that you want indexed!

  • This is good and is exactly what needs to be done!

    Any intel about the performance? Is it gonna change? Is it gonna crawl web sites more often? And more extensively?

  • Great news! sort of . . . I hope they do more than change the name: I hope they actually improve the way it works, providing a more relevant index like that of Google Caffeine.  But why do we have to wait until October?  This mentality of introducing periodic updates every six months or so is so very typical of the mentality of the Microsoft of old: when companies had to send out updates and did so en masse.  What do companies of new like Google do? They come out with updates almost daily, doing web-based updates.  Of course, Microsoft does this, too, with updates to Office, etc. But, seriously, why wait? Why not put out the update bot (and hopefully an updated search) right now?

  • Will the DNS entries for the crawlers also change?

    e.g. 207.46.204.189 currently resolves to: msnbot-207-46-204-189.search.msn.com

    Will it change to something like this?

    bingbot-207-46-204-189.search.bing.com

    thanks

  • Nice information its very useful for my client thnks for sharing

  • Do you think that bingbot will borrow some of the 'features' of yahoobot or will it be a totally new robot made from the ground up? thanks

  • Good news - hopefully the new crawler will be more efficient than the current one.

  • I just logged in to the webmasters tools, and didn't find any pages indexed. What would I need to do in order to get our pages indexed?

  • awesome posting, I'll try to my site here: http://fkmdfc.com

  • thnks for that info

    http://arabesque-store.com