This is a place devoted to giving you deeper insight
into the news, trends, people and technology behind Bing.
Over the past month, I have seen significantly improved results from Bing searches on my site--both on the news and regular web search engines (simultaneously, I've seen better results from Ask.com and Yahoo). But all of a sudden, without making changes, I've run into an issue in which Bing is not indexing my data--and, in fact, says it is unable to do so.
My description presently says "We would like to show you a description here, but the site you’re looking at won't allow us." And when I try to fetch individual pages via BingBot, I get a message about robots.txt blocking the crawl.
The problem is that I do not have any restriction or no-follow on my descriptions (so I don't know why Bing can't pull it), and I don't have any robots.txt restriction that would be blocking BingBot from my article pages.
Site is headlineplanet[dot]com - it's working fine on Google, so there is something exclusive to Bing and those that use similar crawls.
Please let me know if anything pops up as causing the problem - like I said, I've tried deleting robots.txt, putting in very loose robots.txt files, etc -- and nothing has fixed this issue.
can you try in your robots.txt file with following:
I hope it would work, or else your server is giving response timeout.
Hot Bizzle | Social Classifieds
Looks like you have no robots.txt file at all at the moment. I'd strongly recommend to create at least a basic one and then check its visibility for Bing with the "Fetch as Bingbot" tool.
Otherwise, I think that redirecting the whole site to /home/ directory isn't a good idea. Good luck!
Tried this - I am getting the same message from Bingbot (either when I try to fetch the robots.txt file or another page on the site)
Status: The url is blocked by robots.txt.
Actually, even though I still get that error in the Bing Webmaster Tool, my site and news are once again being indexed by Bing, Yahoo and Ask.com, so badhusha's recommendation worked.
Thanks "badhusha" :)
© 2013 Microsoft