Page is a not externally linkable
- WebmasterWorld
-- Content, Writing and Copyright
---- How to prevent scraper sites . . .


mblair - 1:09 am on May 21, 2005 (gmt 0)


digitalv,
I have heard of these spider traps before and have considered implementing one but have a question -- I have read in the Google forums here that some have reported that Google has been actually following links despite being disallowed in robots.txt but then not indexing them. I suppose the same concern could run to Google's mediabot as this might do the same.
In these cases are you whitelisting known Google IP ranges or has this just not been a problem thusfar?


Thread source:: http://www.webmasterworld.com/content_copywriting/1341.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com