Crawl Control

The Crawl Control feature in the Configure My Site Section allows you to exert control over the speed at which Bingbot makes requests for pages and resources on your website. You can tell Bingbot to crawl your site faster or slower than the normal crawl rate for each of the 24 hours in the day. This way you can to limit Bingbot activity when your visitors are on your site and allow us more bandwidth during quieter hours.

If we find a crawl-delay: directive in your robots.txt file then it will take always precedence over the information from this feature.

How to Set a Crawl Pattern Using a Preset

The easiest way to set a crawl pattern that optimizes your bandwidth for your visitors is by using the When do you receive most traffic during the local time day drop-down menu. This drop-down menu provides a number of preset options which you can use to indicate the busiest time on your site based on your local time of day. When selecting any of the presets, we will automatically optimize the crawl pattern by telling Bingbot to crawl more slowly during those hours and faster during the other hours. All you need to do to store the setting is clicking Save at the bottom of the page.

The following example shows a site that chose the 9AM - 5AM local time preset as the busiest time for the site, resulting in an inverse crawl pattern that allows faster crawl speeds for Bingbot outside of those hours:

Crawl Control Hourly Crawl Speed Pattern

How to Set Custom Crawl Pattern

Alternatively, you can use your mouse to create an entirely custom hourly crawl pattern by moving over the cubes holding the left mouse button down and dragging it to the desired number of quadrants for the timeline denoted on the x-axis. More blue quadrants indicate more visits from Bingbot (faster crawl), fewer quadrants mean the opposite (slower crawl). The default is 5 quadrants for each time slice. As a result, 5 blue quadrants represent the algorithmically determined baseline crawl rate for your site. Moving them up or down increases or decreases sets a higher or lower crawl rate based on the baseline crawl rate that was determined automatically for your site. When your happy with your pattern, simply click Save to store the setting.