Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- U of Cal


keyplyr - 11:19 am on Feb 4, 2013 (gmt 0)




As an added comment - I disallow all those copyright check bots & plagiarism bots in robots.txt and AFAIK they've always obeyed.

A bad agent can come from anywhere, and there are lots of script kiddies in campus dorms (don't forget how facebook was born.)


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4541980.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com