Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Matt Cutts asks webmasters: let googlebot crawl js and css


tedster - 4:39 am on Mar 29, 2012 (gmt 0)


It seems pretty straightforward to me. Regular googlebot respects robots.txt directives. If ronots.txt says "don't crawl this URL" then they'll never see any robots meta tag or X-Robots directive associated with that URL.

That's a straightforward technical reality. If you want a robots meta tag or X-robots to be seen, then you've got to let it be crawled. The essence is this:

robots.txt is about crawling
robots meta tag is about indexing


Thread source:: http://www.webmasterworld.com/google/4433745.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com