Crawl Control

The Crawl Control feature allows you to exert control over the speed at which Bingbot makes requests for pages and resources on your website. You can tell Bingbot to crawl your site faster or slower than the normal crawl rate for each of the 24 hours in the day. This way you can limit Bingbot activity on your site when there are more visitors on your site and allow Bingbot to crawl more during quieter hours.

If we find a crawl-delay: directive in your robots.txt file then it will take always precedence over the information from this feature.

How to Set a Crawl Pattern Using a Preset

Use a preset crawl pattern

The easiest way to set a crawl pattern that optimizes your bandwidth for your visitors is by using a preset listed under the “Time when your site receives most of the traffic”. You can use these presets to indicate the busiest time on your site based on your local time of day. When selecting any of the presets, we will automatically optimize the crawl pattern by telling Bingbot to crawl more slowly during those hours and relatively faster during the other hours. All you need to do to store the setting is click Save changes at the bottom of the page.

The following example shows a site that chose the 9AM - 5AM local time preset as the busiest time for the site, resulting in an inverse crawl pattern that allows faster crawl speeds for Bingbot outside of those hours:

Crawl Control Hourly Crawl Speed Pattern

By default, the system time is taken into consideration while determining the time zone. However, If you are working from another country and would like to modify the Bingbot's crawl rate hours based on some other time zone, you can now choose the relevant time zone from the Time zone drop down and modify the crawl pattern accordingly.

Crawl Control Hourly Crawl Speed Pattern

Set Custom Crawl Pattern

Alternatively, you can also use your mouse to create an entirely custom hourly crawl pattern by moving over the cubes holding the left mouse button down and dragging it to the desired number of quadrants for the timeline denoted on the x-axis. More blue quadrants indicate more visits from Bingbot (faster crawl), fewer quadrants mean the opposite (slower crawl). The default is 5 quadrants for each time slice. As a result, 5 blue quadrants represent the algorithmically determined baseline crawl rate for your site. Moving them up or down, increases or decreases the crawl rate based on the baseline crawl rate that was determined automatically for your site. When your happy with your pattern, simply click Save changes to store the setting.