Crawl Information: Understanding Crawl Errors

You can view information about crawl issues that Bing has found on your own site from within Bing Webmaster Tools. This crawl information can help you understand where Bing is having problems accessing your content, or if there are potential issues with your site. This data can be found for your validated websites by clicking on the Reports & Data section in the navigation pane. Click on Crawl Information item to see the data: Crawl Information in Bing Webmaster Tools

In the Crawl Information tool, you can see the various issues we've encountered, organized by issue type. Clicking on the issue type will show a representative set of URLs we found the issue for. You can also export a representative set of examples (up to 2,000 per type) - by clicking Export All.

Issue Types

  • HTTP codes 400-499: If you have pages that return a 400-level HTTP response when we tried to crawl the URL, they will be shown in this particular section. You can a sample of the individual error by clicking on the number shown below the 400 – 499 section. This will show a report in the table below including the URLs that we tried to crawl and the actual response code we are seeing. We will also try to show the count of how many prominent links are pointing to this URL. Note: When the report shows 404 pages, you should check if the URLs are expected to be 404 Not Found. If they are indeed invalid URLs on your site then there is no action necessary on your side: we discover many links on the web and some of them are incorrect, so it is normal to have 404 pages reported and not detrimental to your performance in search. If you find,however, that valid pages are showing up here, you may need to check whether a page may have accidentally been removed. In no event should you try to block 404 URLs in your robots.txt file just to get a clean report in Webmaster Tools.
  • HTTP codes 500-599: If you have pages that return a 500-level HTTP response when we tried to crawl the URL, they would be noted in this section. You can view them by clicking on the number shown below the 500 – 599 section, which will show a report below including the URLs and the actual response code we are seeing, such as a 500, as well as the count of how many links are pointed at the URL.
  • HTTP code 301: If you have pages that return a 301 (moved permanently) HTTP response when we crawled the URL, they would be noted in this section. You can view them by clicking on the number shown below the 301 section, which will show a report below including the URLs and the count of how many links are pointed at the URL.
  • HTTP code 302: If you have pages that return a 302 (moved temporarily) HTTP response when we crawled URL, they would be noted in this section. You can view them by clicking on the number shown below the 302 section, which will show a report below including the URLs and the count of how many links are pointed at the URL.
  • Malware infected: Any page URLs that Bing found that are infected or associated with malware will be grouped in this section. We will have also posted an alert in your Message Center to notify you of the problem, and will sent you an email to your forwarding address if you have that option enabled in your Profile settings.
  • Excluded by robots.txt: All URLs that we are not allowed to crawl according to the rules of your robots.txt file are shown in this section. You should scan this section to ensure your robots.txt file is not blocking content you actually want Bing to crawl.
  • DNS Failures: This issue type catalogs recent errors we encountered trying to communicate with the DNS server when we tried to access your pages. Possibly your server was down, or there was a misconfiguration that prevented DNS routing, for example TTL was set to 0.
  • Connection Errors: This number represents recent occurences when Bing could not access your site due to connection errors. This could be a temporary issue but you should check your server logs to see if you are accidentally dropping requests.

For reference, here is a full list of HTTP response codes and what they mean. Note that we only report the status codes shown in the above list inside the Crawl Information tool.)