Page is a not externally linkable
-- Google SEO News and Discussion
---- Google is spidering everything, including blocked pages
Samizdata - 7:45 pm on Dec 27, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/google/4401750.htm
The introduction of Google Web Preview sounded the death-knell for robots.txt compliance.
It is no consolation that Microsoft bots are even worse.
Only enforcible restrictions seem to work these days.
Brought to you by WebmasterWorld: http://www.webmasterworld.com