In keeping with our themes of rapid change and responding to input and feedback from the webmaster community, we are working on future feature planning for the Bing webmaster tools and we would like to hear your thoughts in a few areas. The first area is Crawl-Delay:
Today, we fully support crawl delay which is specified in robots.txt. We are considering creating a tool which allows you to directly control the frequency, and perhaps even the timing, of msnbot/bingbot. A few questions:
Do you prefer for us to continue to support crawl-delay in robots.txt?
Feel free to reply to this post, or send email to bwmc@microsoft.com.
We thank you in advance for your feedback.
I think its useful to provide webmasters control over crawl frequency. But what really matters (IMHO) is how the content being crawled is actually "indexed" (added to BING). Not only is this important for SEO, but for API usage as well.
A common "quick and dirty" use of BING APIs is to provide a "search engine" for one's own web site(s) - yes, this is pretty much a standard "G" api. A strict comparison between the 2 APIs points me to G simply because I can actually use it based on how fast G crawls content based on submitted sitemaps and makes content searchable. I'm not even talking about SEO/ranking - this is just making all the effort (of using webmaster tools) give a better return....
So hopefully, the larger context of this post is improving the indexing performance....
A tool that can control crawl rate in Webmaster Center is a great idea: I see lot of people in the forum who face the problem of Over Crawling by Bing Bot: some of the users mentioned that they could not stop the bot even with the commands in robots.txt file:
Some even tried blocking via .htaccess:
Example: bing . com/community/Webmaster/f/12252/t/651382.aspx
I think this idea of a tool will be great: However, I personally have never used the crawl delay option: I am quite happy with the way Bing Bot crawls my site but if a tool is available, then I may adjust it according to frequency at which I update my site:
I believe lot of people will love this idea and I will now be waiting for its implementation : )
I agree with the others. I think specifying the crawl-delay is good, but indexing is more crucial. Unfortunately, Bing continues to lag behind Google not only in the rate at which it crawls the web, but the size of its index, and the relevancy of that index. And, for some reason, no one at Bing seems to be doing anything tangible about fixing this. If Bing wants to improve its popularity, it needs to have an index better than that of its competitors; but it does not. Users consistently find the relevancy of Google's searches considerably higher and of better quality and accuracy than that of Bing. Improving this should be a top priority. I've heard people at Bing Webmaster state that this is a top priority, but I see little change in the SERPs. I think the culprit is that MSNbot (soon to be called BingBot) is a terrible job at indexing sites; it consistently fails to index sites, indexes them partially, or too often.
I would like Bing to keep supporting crawl-delay in robots.txt file, but do not wish to control the rate based on time of the day.
Over-crawling + low number of indexed web pages = loss of search market share. I hope you'll get my point.
Well first: Please turn of Crawl delay for non-US websites. This is a ridiculous thing and loads of people here in Europe hate Bing for this. Bing is neglecting their EU customers. Forcing them to use Bing (by putting it in windows and IE) and then giving us total ***.
(for US people that don't now what I am talking about: Outside the US Bing is still Live Search with a Bing logo on it. None of the new features are released here and they crawl like 1 page a week. See: www.remivanbeekum.nl/.../dear-bing-could-you-please-crawl-my-website )
Then your questions:
1. I don't think you guys should focus on robots.txt. Focus on the webmaster tools.
2. I think making an option to crawl more at night (please implement timezones!) could be a good idea. But that would cause problems with the news section. You want to focus crawling on the night, but still get news crawles in a few minutes. So there should be some thing that fixes that. Maybe crawl RSS (with new stuff) fast and crawl archives and pages slow at daytime?
3. You probably set this one time and never look back, unless you make big changes.
4. Focus on the webmaster tools. Forget about robots.txt
Can you clarify one point: if I set a delay of say, 30 seconds in robots.txt, is Bing more likely to SPEED UP its crawling to 30 seconds, or see 30 seconds as a maximum and stick to its regular rate (which I assume is a few minutes)??
I don't think there is any need for a tool in BWT to limit crawl rate but sometimes it's useful to increase crawl rate temporarily if you added 100 new pages. If that can be done with robots.txt then NO to a new tool, otherwise YES.
However, like EdSF I am more concerned with indexing speed. On my site, around 300-500 pages/day are crawled by Bing, according to BWT but only 2,500 out of about 4,000 pages have been indexed. Nearly all have been on the site for 6 months or more.
Scott, crawl-delay of 30 means we will be limited to crawling one document every 30 seconds. It is the max amount we will crawl. Whether we use the ceiling from crawl-delay depends on many other factors.
Bing is very slow at crawling and indexing the web compared to Google. I hardly use bing webmaster tools anymore because regardless of how slick the UI is and what bells and whistles bing webmaster tools contain if the site is not being indexed and crawled so your tools aren't going to do us much good. Personally I use the tools and optimize the site for the search engines that are driving traffic to site. While google tools are not flashy they work well and Google drives 70% of my visitors. You should focus on the core of bing indexing then worry about the features.
I have don everything what bing webmaster tool says, but not get any improvement in my visitor and clicks,
What else i do to improve my visibility with Bingh Search Engine?
I am having countless issues with getting my sites re-indexed. I was very well ranked for a handful of sites. Around Oct-15, my sites disappeared from the bing and yahoo index. I have re-submitted them but no luck ... they are still not indexed.
I think 'boardofthings' has a point there. There is no use in building some kind of crawl delay when you already crawl like a snale.
So please stop developing any new fancy stuff and start indexing the internet!
Please forgive my ignorance. But what is crawl delay? For some pages, I would like to let Bing know to come back every day (say for a blog). Is that what you mean?
I would like Bing to keep supporting crawl-delay in robots.txt file, Great
I would like to see this feature fully implemented. Being that most of my humans visitors visit during morning / day / evening hours it would be great to tell Bing to index the site at night when traffic is low.
thank you admin
