Bing blogs

This is a place devoted to giving you deeper insight
into the news, trends, people and technology behind Bing.

Webmaster Blog

April
18

More crawling improvements from MSNBot

A few months ago we announced two new features to MSNBot to reduce the burden of crawling on your website. These were part of a series of improvements we’re making to our crawler during the Spring to increase the freshness and breadth of content in our index. As part of these latest improvements, you may notice an increase in the amount of traffic from MSNBot starting over the next couple weeks. If you notice any issues with MSNBot, please make sure to drop us a note on our Crawling Feedback & Discussion Forum so we can investigate.

This is a great time to take a look at your robots.txt file (and meta tags) to make sure that you are not inadvertently blocking robots from content on your site you may want indexed. Also, if you feel that MSNBot is crawling your site too frequently, you can use the crawl delay directive in robots.txt. Please refer to the MSNBot support page for more information. Here are a few recommended settings:

Slow (wait 5 seconds between each request)

Crawl-delay: 5

Really Slow (wait 10 seconds between each request)

Crawl-delay:  10

Note that setting the crawl delay reduces the load on your servers, but it also increases the amount of time it will take MSNBot to index your website (proportional to the length of the delay), and possibly make it more difficult for your customers to find your site on Live Search.

Another great way to reduce the impact of MSNBot on your website is to enable HTTP Conditional GET and HTTP Compression as outlined in our prior blog post.

--Nathan Buggia, Live Search Webmaster Center

Comments

  • "you can use the crawl delay directive in robots.txt. "

    This is a useful fearture.

    We could use bandwidth efficiently.

    I wish if other search engines would support the same fearture.

  • Hi,

    Microsoft Live Search had not crawled my site for 3 months. Meanwhile Google and Yahoo had and I get visitors from Microsoft employees who use Google at work. Anyways, I removed the GZIP compression I was using and Microsoft Live now manages to get through somehow.

    Every browser and every other bot could deal with the compression. Seems like a bug in Live Search. Contact me at smallenucd@yahoo.com if you need me to help debug the situation. Thanks

    Sam

  • Yahoo also supports this feature, but you'll need to use the Google Webmaster Tools if you would like to ask Google to slow down their crawling.

  • I am glad to see that you've increased crawl rates, but I'd like to point out the lag between MSN and other search engines.

    * Google bot is on my site every few minutes, pretty much 24/7.

    * Yahoo is on my site a few times an hour.

    * MSN Bot- maybe once or twice a month.

    This is with a robots.txt discoverable valid sitemap containing 20,000+ pages (within protocol).

    I submitted my site to your 3 engines at the same time, and for the first time, last November 07.  Not surprisingly, Google is the only engine to have indexed most of my site, and where >90% of my visitors come from.

    I am using asp.net on IIS6 gzip enabled, conditonal get off, no code errors ever reported when MSN does crawl.

    I hope MSN doens't always stay this slow.

  • Really happy to see the improvement, hope MSN Bot can come to my site more often. Most of my site are not so well indexed by MSN, hopefully after the implementation my site will get more attention from MSN.

  • My website disappeared from your Search result for 2 years causing me lot of money loss. If you can't run a search engine please close it.

    Please give the address of the mathematician developed the algorithm for you. I want to sent 5 cents as my reward for inventing most crappy algorithm in the world.

    Regards.

    365greetings.com

  • Now have passed 2 months and my site www.claudio-corti.com has not been spidered yet :/ pretty sad

  • Crawling slowliness is actually one of actual search engines failures. Any web-editor cant wait 5or more days to see one result

  • Currently only my main page is crawl, let see whether this improvement will lead to other of my web page crawled.

  • Although my home pages is index, but it only index 1/100 of my site.

  • Sam here, I started my blog three months ago and only the main page has been indexed while Google and Yahoo have indexed every article within a few days. Google even within one hour after publishing it.

  • Oh, I can control craw time of MSNBot! Great support!

  • My site is index in MSN search but is there a way to find the rank of your site?

  • great news, i should pay more attions about msn bot now.

  • This blog has been created to share useful information. Thanks and greetings! athough msn only index some of my site.