Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- And Now Google's Doing It. JS Stats Show GoogleBot


Samizdata - 11:19 pm on May 14, 2011 (gmt 0)


they can throw protocol to the wind when it suits their purposes

That sums it up - if you don't want Google to take your files, don't put them on the web.

But you want the Faustian pact to work for you, and keyplyr had it right:

I only fret when they're indexed

...


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4312058.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com