List of Crawl Error Alerts


The table below describes the various Crawl Error alerts that we send to the Message Center when we see a significant or significantly increasing amount of issues when crawling your site.

 

Issue Category Specific Type Alert Action Message Comment
400-499

401 (Unauthorized)

We encountered a large number of request that returned the following status: 401 (Unauthorized)

Your server is refusing access to your pages and requires additional information, such as a login and password. Check if you are unintentionally blocking your pages behind a login and password. If this is intentional, you could consider adding a disallow: rule to your robots.txt so that we no longer attempt to crawl your pages. When your server responds to our request with a status code of 401 (Unauthorized), this likely means we are trying to crawl an area of your site that is password protected and for which we would require additional authentication. Check some sample URLs for this error in the Crawl Information tool to see if this is expected or if you accidentally walled off content in a password protected area. If this is expected, you could consider adding a so-called disallow: directive to your robots.txt file so that we don't spend time (and bandwidth) crawling these protected URLs going forward.
400-499

403 (Forbidden)

We encountered a large number of request that returned the following status: 403 (Forbidden)

Your server is denying access to your pages to Bingbot. Check if you are unintentionally blocking some or all Bingbot requests on your server. Your server may have been configured to only accept requests from Bingbot from specific IP ranges. Since Bingbot IP ranges may change over time, you may be accidentally blocking valid Bingbot requests this way. You should check your server code to see if this is the case. To see whether Bingbot requests are really coming from Bing machines instead of suspicious crawlers that are spoofing the Bingbot user agent, use the methods described in How to Verify Bingbot. Note that you can always use robots.txt directives to prevent Bingbot from crawling sections of your site. If you feel we are crawling you too much say during business hours, consider setting hourly crawl rates using the Crawl Control tool instead of blocking requests this way.
500-599

5xx (aggregate of all server errors)

We encountered an increase in the number of requests that resulted in a server error. Your server reported a large number of server errors and wasn't able to fulfill our requests. This may be caused by a temporary issue, but could also point to a code error or a misconfiguration on the server. This alert triggers if we've seen a significant increase in server errors overall. The reasons for these issues may vary, but you should check your server logs to see if the issues are systemic.
500-599

500 (Internal Server Error)

We encountered a large number of requests that returned the following status: 500 (Internal Server Error) Your server reported an internal server error and wasn't able to fulfill our request. This could point to a problem in your site's code or misconfiguration. This response code generally indicates that there was a problem on the server without more specifics. It can point to a code issue or a configuration issue. You should check a few of the URLs that suffered from this issue in the Crawl Information tool and check the server logs for additional clues.
500-599

503 (Service Unavailable)

We encountered a large number of requests that returned the following status: 503 (Service Unavailable) Your server reported that it was not able to handle our requests. This could have been a temporary issue, but you check your logs and scripts for performance-related issues. This response code could have been issued because the server was unavailable. Perhaps it was overloaded or down for maintenance at the time. This should be a temporary problem but you should check the samples in the Crawl Information tool (see the 500-599 section) to see if the problem still persists.
500-599

509 (Bandwidth Exceeded)

We encountered a large number of requests that returned the following status: 509 (Bandwidth exceeded) Your server reported that bandwidth was exceeded and it was not able to handle our request. Please check your server configuration. When your server responds to requests with HTTP Status 509 (Bandwidth Exceeded) could point to throughput issues or your server may be restricting the bandwidth that Bingbot is allowed to consume. Not addressing this could result in comprehensiveness and freshness issues. A recommended way to control the speed at which Bingbot crawls your site on an hour-by-hour basis is to use the Crawl Control function inside Webmaster Tools.
Crawl politeness Crawl delay setting Your crawl delay setting in robots.txt is preventing Bing from crawling effectively. Please remove the crawl delay setting from your robots.txt. In addition, please make sure you have an optimal crawl rate setting configured on the Bing Webmaster Tools crawl settings page.
Bing honors the crawl-delay directive in your robots.txt file. If the crawl-delay you have specified prevents us from crawling your site efficiently you will receive this alert. If possible, remove the crawl-delay directive altogether and control crawl behavior using the Crawl Control function inside Webmaster Tools instead. 
Crawl politeness Increase crawl rate The current crawl setting for your site is preventing Bing from crawling effectively. Please make sure you have increased crawl rate setting configured on the Bing Webmaster Tools crawl settings page.
This alert means that we would like to crawl your site more efficiently but that your current crawl control settings are preventing us from doing so. You can increase crawl speed sing the Crawl Control function inside Webmaster Tools.
DNS Issues Large number of DNS Failures We encountered a large number of DNS failures when trying to crawl your site. When attempting to crawl your site we encountered DNS Failures. These may be the result of a server or DNS misconfiguration. In some cases, the DNS server responsible for you website may have been misconfigured. A common issue is a TTL (time-to-live) setting that is set to 0. In other cases, the DNS service may be filtering out requests from Bingbot. You might be able to check this using Fetch as Bingbot which will tell you if it cannot resolve the DNS.
Connection Issues Large number of connection issues Bingbot could not access your site due to connection errors. This could be a temporary issue but you should check your server logs to see if you are dropping requests. In some cases, your server may start stop responding when it is overloaded. To make sure you are not actively refusing connections for requests from Bingbot, use the Fetch as Bingbot.