I'm having a bit of a problem because I get a lot of visits from this crawler-acting-like user that has a UserAgent String of "Mozilla/4.0" and that has the same range of ip adresses as MSNBot. Indeed the host name for this IP adresses is MSNbot.
Knowing all the bad reputation that MSNbot used to have in its origins, are they testing a new bot, or is just that knowing that so many sites blocked MSNBot for being invasive and band-width agressive, they change the UserAgent String so that the robots.txt won't affect them so they can bring Bing up to somewhere?
That was a long question.
Hi,
We've received word from a few other sources concerning this. Our team is reviewing it and will let me know as soon as they have an update. Please be assured that it is not our intention to misrepresent the bot or cause undo problems for webmasters.
~B
*I no longer work for Bing.
Hi Brett,
What's the status of this? I'm seeing msnbot blocked by "Bad Behavior" because it's reporting as Mozilla/4.0. It's been nearly a month since this problem was reported (via this thread) and it's obviously still occurring.
Thanks!
It sounds too strange to me that MSNBot is crawling masking its Useragent name as Mozilla/4.0. Well, I haven't noticed that in my log stats yet. I hope this issue gets sorted out quickly.
The best web directory | Computer products | SEO Blog | PHP Scripts
We've also noticed this… The most strange thing was that Netscape was the most used browser with version 4.0 in our AWStats :D
Those requests came from 65.55.*.* IP range and started at October 31 and ended at November 10 giving us around 3million page requests per day.
Brett Yount: Our team is reviewing it and will let me know as soon as they have an update.
Our team is reviewing it and will let me know as soon as they have an update.
I'm still getting regular hits from the msnbot IP's with a "Mozilla/4.0" user agent. Seeing as this was reported over two months ago it doesn't seem like anybody is interested in finding out what is wrong?
Unfortunately I'm going to block msnbot completely until this is fixed.
Mozilla is the way to go.
Fence Atlanta
Fence Materials
I am facing the same issue for the past one month...
NAPW’s MerchantCircle Page; Vator’s NAPW
This problem is getting more serious. Many people are facing the same issue.
Divorce Attorney Atlanta
Divorce Attorney Athens
hope this issue gets solved fast.
Cash For Gold
Gold Party
I believe this behavior is intentional. The bot acts like a normal visitor, even downloading js and css files.
For those that are worried about MSNBot consuming a lot of their bandwidth, you can use crawl delay to set the crawl rate for the bot. See the following blog posts for information on how to do it: http://www.bing.com/community/blogs/webmaster/archive/2009/08/21/prevent-a-bot-from-getting-lost-in-space-sem-101.aspx and http://www.bing.com/community/blogs/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx
I'm sure it's intended by MS... but it's inappropriate for a search engine to look like anything OTHER than a search engine. Google doesn't do this kind of crap.
All that said... if this was really the way they wanted it, then MS should just come out and say that... rather than ignoring this issue.
that is not what this thread is about. It's about the webcrawler (MSNBot) pretending to be Mozilla.