Verifying BingbotIf you see what appears to be Bingbot traffic in your server logs based on a user agent string, for example Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm), and you want to know if this traffic really is originating from a Bing server, you can take the following steps:
- Perform a reverse DNS lookup using the IP address from the logs to verify that it resolves to a name that end with search.msn.com
- Do a forward DNS lookup using the name from step 1 to confirm that it resolves back to the same IP address
There are ways to do this directly from within Windows and Linux, but you can use the Verify Bingbot tool right from within your Webmaster account or use the publicly available version at http://www.bing.com/toolbox/verify-bingbot. Alternatively, you can use other web-based tools:
Web-Based Reverse DNS & IP Lookup ToolsInstead of using command-line tools supplied by your operating system, you can use one of the many web-based reverse DNS lookup tools. Here’s an example of a reverse DNS lookup for IP address 18.104.22.168 using http://www.whois.net/reverse-dns-ip-lookup/:
Reverse & Forward DNS Lookup in WindowsOn Windows systems, you can use nslookup from the command prompt (using cmd.exe) for the both the reverse and forward DNS lookup. Here is the example of the reverse lookup for IP address 22.214.171.124; look for the line starting with Name: in the output. This is where you want to find the name that ends with search.msn.com:
C:\Users>nslookup 126.96.36.199 Server: Unknown Address:
Now do the forward DNS lookup; the output in the second Address: line should match the IP address you entered before to confirm it’s a Bing crawler:
Name: msnbot-157-55-33-18.search.msn.com Address: 188.8.131.52
C:\Users>nslookup msnbot-157-55-33-18.search.msn.com Server: UnKnown Address:
Non-authoritative answer: Name: msnbot-157-55-33-18.search.msn.com Address: 184.108.40.206
The verdict is the same: the name and the address match, so this is a verified Bing crawler.
Reverse & Forward DNS Lookup on Linux-based systemsOn Linux you can use the host command to do the same:
> host 220.127.116.11 18.104.22.168.in-addr.arpa domain name pointer msnbot-157-55-33-18.search.msn.com > host msnbot-157-55-33-18.search.msn.com msnbot-157-55-33-18.search.msn.com has address 22.214.171.124
Don’t Use Hardcoded IP Addresses or Address Ranges
So, by using the reverse/forward DNS lookup method you can easily verify that an IP address is coming from Bing. It is important to note that like other search engines, Bing does not publish a list of IP addresses or ranges from which we crawl the Internet. The reason is simple: the IP addresses or ranges we use can change any time, so responding to requests differently based on a hardcoded list is not a recommended approach and may cause problems down the line.
If you are experiencing an increase in HTTP 403 Forbidden responses to valid Bingbot requests, your webserver may have been configured to allow Bingbot access based on such a list. As a result, new Bingbot crawl machines (with new IP addresses) may unintentionally be denied access to your pages.