Today we're pleased to announce an update to the Sitemaps Protocol, in collaboration with Google, and Yahoo! This update should help many new sites adopt the protocol by increasing our flexibility on where Sitemaps are hosted.
Essentially, the change allows a webmaster to store their Sitemap files just about anywhere, using a reference in the Robots.txt file to establish a trusted relationship between the Sitemap file and the domain or folder.
Here's how it works: Say you run a web site like MSN.com, which has a bunch of sub domains like health.msn.com, travel.msn.com and moneycentral.msn.com. And, due to a technical requirement, you would like to host all of your Sitemaps in one location like sitemaps.msn.com. Until now the protocol did not support this scenario, each Sitemap would have needed to be hosted directly under the domain it described. This update now introduces support for this scenario, with the requirement that you simply include a reference to the Sitemap in your Robots.txt file. For example, moneycentral.msn.com/robots.txt would need to include this line:
Sitemap: http://sitemaps.msn.com/index_moneycentral.msn.com.xml
The catch is that all the URLs in the Sitemap file all need to be within the same domain as the robots.txt file (i.e. moneycentral.msn.com/* in this example). Note that this applies equally for Sitemap index files and for compressed files.
Here are a few other useful notes about our implementation:
This change comes directly from feedback we received from webmasters, thank you for helping us improve our product! If you have any additional feedback or questions, please check out our Sitemap Discussion forum.
--Fabrice Canel, Program Manager, Live Search Crawler
