Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- What's the Best Way to Keep All Spiders/Bots Out?


Seb7 - 11:10 pm on Oct 3, 2011 (gmt 0)


I would use a combination of tricks, with honeybot to ban bots by IP.

Something I just thought of, is to have a anonymous login page, not a page that requires a password, but just test to see if they are a real user. If real, then allow that session to view the rest of the pages.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4369764.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com