Hi - we get a ton of requests from msnbot/2.0b for a URL that is Disallowed by our robots.txt file. Any idea why that is?
Hi Drake,
received your email. I'm forwarding it to the Engineers for review. I'll let you know as soon as I hear back.
@All
If you are having msnbot 2.0 issues, please email us at bwmc@microsoft.com and we will investigate ASAP.
Thanks,
~B
*I no longer work for Bing.
Hello Brett,
Our Robots.txt had the basic User-agent: * to allow all bots to crawl our site, but we have not seen any recognition from Bing. So we added a duplicate commands with User-agent: msnbot thinking maybe this was needed to specifically address msn bots. Well, no results yet at webmaster central that we know.
Should disallowed content display in the Crawl Issues section under Blocked By REP?
Thank you