We've already covered in past blog articles some of the basics about how webmasters can use a file called robots.txt to control how search engine crawlers (aka bots) crawl their websites. But there is so much more to talk about with bots. So let's take a bit of a deeper dive into the subject. Topic 1...

The Bing team has been talking about its new crawler (aka bot), MSNBot 2.0b, in this blog for quite some time now. We have made numerous improvements in its performance, addressed some webmaster concerns, and published detailed information on how to control the bot with a robots.txt file . Today we are...

We've been really busy here at the Bing Webmaster Center blog team, pumping out new content on a regular basis to create a nice library of content on issues that matter to webmasters and online publishers . I thought I'd take a moment to catch my breath, pause on creating a new thematic article (or yet...

There’s been some recent discussion in the SEO blogosphere asserting that Bing clicks its own adCenter ads. This has created some misunderstanding. Let’s take a moment to clarify what is actually happening, and what this really means for webmasters and advertisers. The Bing team is aware of an issue...

We recently published a non-SEM 101 blog post on controlling the crawl rate of MSNBot , the Bing web crawler (aka robot, or simply just bot). That got me thinking about robots. Naturally, that led to The Robot on Lost in Space . Will Robinson , the show’s precocious youngster who was a whiz at 1960s...

Search engines, such as Bing, need to regularly crawl websites not only  to  index new content, but also to check for content changes and removed content. Bing offers webmasters the ability to slow down the crawl rate to accommodate web server load issues. The use of such a setting is not always...

Back in December of 2008, we alerted you to the possibility that you might start seeing our new web crawler in your referrer logs. Given the recent excitement around the arrival of Bing, we wanted to take a moment to update you on the latest bot news. As we said earlier, you will see the new Bing crawler...

In a blog post last month , we identified the preferred method for providing feedback on any Live Search crawler issues you may have experienced. We would like to thank everyone for taking the time to provide us with this feedback and let you know what we have done on our end. But first, a little background...

Just recently a strange problem came across my desk that I thought was worth sharing with you. A customer notified us that content from a site she was interested in was not showing up in our results. Wanting to understand why we may or may not have indexed the site, I took a look to see what the problem...

My favorite feature of our recent launch is the Crawl Issues tool, which gives you details about issues Live Search may encounter while crawling and indexing your website. This information can help you better understand what Live Search sees when crawling your site and should ultimately help you improve...

As a member of the Live Search Webmaster Team , I'm often asked by web publishers how they can control the way search engines access and display their content. The de-facto standard for managing this is the Robots Exclusion Protocol (REP) introduced back in the early 1990's. Over the years, the REP has...

A few months ago we announced two new features to MSNBot to reduce the burden of crawling on your website. These were part of a series of improvements we’re making to our crawler during the Spring to increase the freshness and breadth of content in our index. As part of these latest improvements...

Today we're pleased to announce several improvements in the crawler for Live Search that should significantly improve the efficiency with which we crawl and index your web sites. We are always looking for ways to help webmasters, and we hope these features take us a few more steps in the right direction...