I've had this robots.txt file on my server (http://www.franklin.library.upenn.edu/robots.txt) since at least January 2009.
User-agent: *
Disallow: /
Hi McKenzie,
We are tracking this situation and vigorously working to fix these errors. Could you please send an email to bwmc@microsoft.com with your domain name and the title of this post in subject line. Could you please also send any documentation, such as clips from your log file, that might help us positively identify which bot is causing this issue.
Your help is greatly appreciated.
~B
*I no longer work for Bing.
We are having the same issue on our site.
Our site contains forums, and the MSNBot is crawling the forum pages sometimes 4 or 5 times in the same day, with multiple connections every second. We have updated the robots.txt file per Bing specs, but nothing has changed.
We are not having bandwith issues, but the forums are database driven and at times the database is bogged down with requests.
Other than blocking the IP address of the bot, is there any other way to get the bot to obey the robots.txt limitations?