In another installment of the From the Forums column, today we have an interesting thread about a website that updated its content and used 301 redirects to migrate users to the new content. However, the transition to the new content in the Bing index was taking longer than they expected. The old content was not fading away.
The amazing volunteer members of our community jumped right in and began examining the site in question. It was soon uncovered that the 301 redirects implemented by the site were also applied to a custom 404 error message page, which itself provided search engines with a 200 OK code, thereby indicating that everything is good (instead of the typical 404 File Not Found error, which would facilitate the removal of those page listings from the index). Brett Yount, our forums administrator, did a quick check with the Bing engineering team, who confirmed that this was the likely reason the old content remained in the Bing index. The webmaster for the site is now looking into the matter with his web team with specific resolution advice in hand for resolving this situation.
This is an interesting case of having a content migration plan in place and demonstrating wise patience for changes to propagate through the system. But when an undiscovered error in the migration plan's implementation occurred, thereby preventing the desired result with search, the webmaster came to the Webmaster Center forum to get the advice they needed to resolve the issue. Check out Again....301 Redirect issue with Bing...going crazy here to see the whole conversation.
If you have any questions, comments, or suggestions, feel free to post them in our Webmaster Center forums. Later...
-- Rick DeJarnette, Bing Webmaster Center
"a custom 404 error message page, which itself provided search engines with a 200 OK code"
this is a common mistake !
I thought spiders didn't rely on these custom 404 pages, they should always fetch the page themselves, otherwise this leaves room for manipulation by blackhat seos.
Credittop,
Further investigation of this situation by Brett Yount, our Webmaster Center forums administrator, uncovered additional details that ultimately were the cause of this webmaster’s concerns. The key finding specifically revolved around the configuration of the site’s .htaccess file. As a result, the initial assessment of the problem as a soft 404 issue has been revised and the problem has been addressed in off-line conversations between Brett and the site’s webmaster. Nevertheless, your excellent question about how search engines work with custom 404 pages deserves to be addressed, so I’ll do that here.
Search crawlers do not rely on the mere presence of custom 404 pages (although these are very helpful assets for human visitors!) for determining whether or not to keep a page in the index. Instead, they rely on returned HTTP codes.
If a page returns a 301 code, that permanently redirects the bot to the page’s new location. If a page returns a 404 code, the bot recognizes that the URL is not valid, thus that URL can be safely purged from the index. Pages that return a 200 code are active and no changes are needed in the index.
When a page associated with a formerly working and indexed URL returns a 301 redirect specifically to a custom 404 page, and that custom 404 page itself returns a 200 OK code, the bot interprets the custom 404 page to be the new, permanent destination of that original URL and assigns all previous index rank status to that “new” page. And as a result, the old URL stays in the index because the redirect refers to a valid page. However, when a webmaster wants old pages purged from the index and uses a 301 pointing to a custom 404 page, that custom 404 page should return a 404 code, which will tell the bot that the original URL is no longer valid.
Sorry if this was not clear in the post! Thanks for writing!
Rick DeJarnette
Bing Webmaster Center team
What is the actual code in the page that tells a bot it's a 404 page?
Is it: '<body class="error404 logged-in">' ?
Or just 'Error 404 - Not Found' text?
-POTG
www.pricesontheground.com
POTG,
HTTP response codes are a function of the web server used to host a site and how it is configured rather than any embedded coding within a webpage. To see the HTTP response code from a given webpage, use one of the many free HTTP tool add-ins for Firefox or use a tool like SEO-Browser (in advanced mode) to browse a webpage.
Designating a webpage as a custom 404 error page is also a function of the web server. You can create a custom page to show whenever the page requested from the web server is not found and associate that page with the 404 error condition. IIS servers do this through the server config interface, while Apache servers do this through editing the .htaccess config file. For more information on creating custom 404 error pages, see the blog post Fixing 404 File Not Found frustrations (SEM 101).
Thanks for writing!
Are inbound links recovered when a 301 redirect is used. Thanks
240 Microsoft Team blogs searched, 101 blogs have new articles. 288 new articles found searching from
Damn my 404 returns a code 200 :-(
Thanks for pointing out this problem. I'll proceed to fix my 404.
Oh, I did not know that. Also, my 404 returns a 200 code as header.
In this blog, we've previously discussed matters relating to custom 404 pages in Fixing 404 File Not
Here is how you preperly set up a 301 permanent redirect to a error page that returns the proper header of "404 Not Found"
Example:
Say on my site you enter this page (which doesnt exist), it will 301 to my 404 page and return a true 404 (not a 200 OK)
www.havocarcade.com/.../1002.html
add this to the top of your custom error page:
<?php
header("HTTP/1.1 404 Not Found");
?>
DONE.
Now your cutom error page will return a true 404 instead of a 200
I never knew this either.
