We've already covered in past blog articles some of the basics about how webmasters can use a file called robots.txt to control how search engine crawlers (aka bots) crawl their websites. But there is so much more to talk about with bots. So let's take a bit of a deeper dive into the subject. Topic 1: Using the proper text file encoding The robots.txt file is used by webmasters to either specifically define which files and directories that compliant search engine bots may or may not crawl. Robots.txt files are basically text files . However, even something as seemingly straightforward as a text file is not as simple as it might seem. Which type of file encoding scheme is used to save the file...
The Bing team has been talking about its new crawler (aka bot), MSNBot 2.0b, in this blog for quite some time now. We have made numerous improvements in its performance, addressed some webmaster concerns, and published detailed information on how to control the bot with a robots.txt file . Today we are announcing that the new bot is fully operational. This development will enable Bing to do a better job at gathering the information we need from the myriad of websites we index worldwide. As MSNBot 2.0b enters full-scale production, the time has come to retire our previous generation bot, MSNBot 1.1. By the end of the first week in November, you will no longer see the following user agent in your...
You've seen it. So have I. Nearly every person who has actively browsed the Web for more than 15 minutes has seen it. I'm talking about the dreaded 404 File Not Found error. When it occurs, users simply abandon their search on that site and go elsewhere. That's a potential lost sale, subscription, or download opportunity (aka conversion) for the affected site! It has been estimated that up to 10% of traffic to large websites on the Web is looking for pages that don't exist, so this is a big problem. Back in the early days of the Web, when you entered an incorrect URL, you got a nearly blank, stark white screen containing nothing but the simple words, "404 File not found." Yeah, thanks. That's...
About eight months back , the Microsoft Research Translator team delivered an entirely unique way of delivering your website's pages to visitors who speak a different language with no development effort on your part. Unlike any other translation widget/gadget available at that time, the Translator widget was unique in that it kept your audience on your site, rather than redirecting them to a proxy translation service. Since then, thousands of sites have adopted the Translator widget and have been able to attract a much broader audience from around the world. Powered by the same machine translation technology that is used by Bing , Internet Explorer , and Office , the Translator widget provides...
We've been really busy here at the Bing Webmaster Center blog team, pumping out new content on a regular basis to create a nice library of content on issues that matter to webmasters and online publishers . I thought I'd take a moment to catch my breath, pause on creating a new thematic article (or yet another multi-part series!) for SEM 101, and address some commonly asked questions in the blog comments. Q: Why wasn't my question in the blog comments answered? A: Well, the Webmaster Center blog is really not the right place for such back and forth exchanges. In fact, at the end of each blog post, there is a reminder to post your questions and comments in the SEM forum . The collection of Webmaster...
OK, so I totally geeked out with my recommendations on how to better secure your webmaster computing environment. As a result, I had too much material for one post and thus had to split it up into two pieces. Let’s wrap up this long series of posts on malware by finishing up with the last of the security recommendations. In Part 1 of this series on malware , we discussed how to detect a malware infection on your website using tools like Bing’s Webmaster Center . The Part 2 post covered the resources and strategies for identifying the types and locations of malware code that typically affect websites with advice on how to remove it. The Part 3 post began the run-down through 10 recommendations...
We’re going to diverge a bit from our regularly scheduled programming. Normally this column discusses search engine optimization (SEO) and related elements of search engine marketing (SEM), but we’re knee deep into our multi-part series on malware and we’re going to begin the wrap-up with a talk about improving computer security. However, I geeked out a bit here, and the column went a bit long (yeah, even longer than usual!), so I decided to break this last section up into two pieces. Who wants to read a white paper as a blog post? I mean, besides me? :-) While beefing up your computer security practices won’t necessarily have a direct affect your site’s SEO performance, consider the repercussions...
Malware infections are no laughing matter. When they afflict your website, they can infect your customers, who won’t appreciate your sharing, intentional or not (and I’m guessing it’s not)! And if Bing discovers malware on your site, your listing in the Bing search engine results pages (SERPs) will either be completely omitted or the link to your site will be disabled, so when the searcher clicks on it, only a malware warning appears. All told, this is bad news for conversions, don’t you think? This article is Part 2 of a three-part series on malware. Part 1 covered how to detect the presence of malware on your site by using the Bing Webmaster Center tools to get access to the information the...
As some of you may have experienced, our “Add a site” page has been unavailable in some locales for the past several days. This is not a site-wide or even a global issue. Unfortunately, this happened due to an update that uncovered a bug in the original code, which caused us to disable the page in one of our data centers. The good news is that we are testing a fix now and will be releasing it to production just as soon as we are sure it will not cause further complications. I will update this post when I receive word that the fix is in production. On behalf of all of us here in the Bing Webmaster Center team, we would like to extend our apologies and thank you for your continued patience...
The Web is an incredible place, filled with amazing media, fascinating content, and wonderful social opportunities, and there’s more of each than anyone can possibly ever consume. But unfortunately, it’s not a benign place. There are more than a few malefactors out there who actively seek to take over your computer for a variety of nefarious purposes. These purposes usually include turning your computer into a: Member of their computer zombie army, available on command (and to the highest bidder) to execute massive distributed denial-of-service (DDOS) attacks on other web-based computers Recorder of keystrokes so they can steal passwords to users’ online financial accounts,...
When I attended both SMX Advanced in Seattle back in June and SES San Jose just a couple of weeks ago, I heard a lot of questions from webmasters about Bing, especially pertaining to search engine optimization issues. Typically these included: I want to do SEO for Bing—where should I start? How is Bing different in terms of SEO? What do webmasters need to know and do? Are there any insider tips for successful ranking? I’ll tackle these questions by providing some useful, baseline information and include pointers to more detailed, pertinent docs. As you know, Bing is an evolution in the search engine space. With its innovative, new user interface (UI) design bringing new depth and opportunities...
There’s been some recent discussion in the SEO blogosphere asserting that Bing clicks its own adCenter ads. This has created some misunderstanding. Let’s take a moment to clarify what is actually happening, and what this really means for webmasters and advertisers. The Bing team is aware of an issue shared by all search engines: paid advertising links on sites are, on occasion, crawled and indexed by search engines. Standard practice in the search industry is to scan web pages for the purpose of indexing and understanding the site’s content, and to determine which ads match best the destination site. Microsoft adCenter does not charge an advertiser for clicks generated by any known search engine...
We recently published a non-SEM 101 blog post on controlling the crawl rate of MSNBot , the Bing web crawler (aka robot, or simply just bot). That got me thinking about robots. Naturally, that led to The Robot on Lost in Space . Will Robinson , the show’s precocious youngster who was a whiz at 1960s-style, clunky electronics (even though the show was supposedly set in 1997!), was best friends with The Robot. They looked out for each other and helped each other in times of need. In a way, search engine bots and webmasters have a similar relationship. They need one another. Webmasters need search engine bots to crawl the pages of their sites so that they can be added to their indexes. Bots need...
We recently published a popular blog post called Get detailed site analysis to solve problems that highlights the function and capabilities of the new, beta search engine optimization (SEO) Toolkit from the Microsoft Internet Information Server (IIS) team. The tool works as an extension to the latest version of IIS, version 7. We’ve received some webmaster feedback with specific questions on how to set up IIS 7 before installing the IIS SEO Toolkit. So to help simplify and clarify this task for webmasters (as clearly folks are very much interested in using the tool!), we developed this quick “How-to” post with detailed instructions for the set up of IIS 7. We hope this helps...
Have you ever noticed how pirate treasure maps are like Sitemaps? While your website may not contain a treasure of gold and silver (unless it’s a metals commodities trading site!), if you have good content, that is certainly treasure to someone who is looking for it. Unfortunately, it’s buried on your website and no one knows what’s there except you! But since you want to share your site’s treasure with others, you need to let them know what you have buried and where to find it. You can wait for search engine crawlers (aka bots) and random traffic to come by to browse, but that will take time and even then, they might not discover everything that you have to offer. Instead...
