Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Googlebot possibly ignoring robots.txt

lucy24 - 11:15 pm on Apr 26, 2012 (gmt 0)


... but once all of that's taken care of, you still have the cases where, for example,

Disallow: /piwik

has to be supplemented with

RewriteCond %{REMOTE_ADDR} ^(207\.46|157\.5[4-9]|157\.60|209\.8[45])\. [OR]
RewriteCond %{HTTP_USER_AGENT} (Bluecoat|Bot|facebook|Google|Preview) [NC]
RewriteRule piwik\.(js|php)$ - [F]

Thread source:: http://www.webmasterworld.com/robots_txt/4446154.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com