Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Matt Cutts asks webmasters: let googlebot crawl js and css


lucy24 - 4:37 am on Mar 27, 2012 (gmt 0)


If g### doesn't like what it sees in robots.txt why doesn't it just ignore it? Preview, Translate, Goggles and Wireless Transcoder already do. So does the plainclothes msnbot and everything from Yahoo.

I've got certain well-known robots blocked from piwik.js via htaccess because they refuse to honor instructions to keep the ### out. What do they think-- that instead of a well-known analytics program, I've got a secret stash of extra formatting in there?


Thread source:: http://www.webmasterworld.com/google/4433745.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com