This is a place devoted to giving you deeper insight
into the news, trends, people and technology behind Bing.
my site http://youthdimension.com doesn't have robots.txt file by this mean i want search engine to crawl my entire site. But recently when i submitted site to various search engine they said missing robots.txt can't able to crawl. And also please tell me how to get adsense to fast and how to increase website traffic
If Your Site not indexed by search engines or you want to change the meta description then you need robot.txt file otherwise sitemap is enough
My Site : webseotrix.net
It gives you great control. You might want to create one. It doesn't mean, search engines won't crawl your site but it's really useful.
My site createawebsite4free.com has no robots.txt but still google, yahoo and bing are Indexing my website including all my post. The real truth about getting a site Indexed is getting links pointing to it.
So stop thinking about robots.txt and think about creating phenominal content, which people would love to talk and share about.
Share your post face book and ask your friends to share and like it. It's gonna build you links and your website will get indexed.
Robots.txt file does not mean search engine index your website. Robots files allows webmaster to which page index or which not to submit search engines.
© 2013 Microsoft