Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Lots of interesting user agents


lucy24 - 7:30 am on Nov 23, 2012 (gmt 0)


Hee. That list looks like my BrowserMatch list, only longer.

:: shuffling papers ::

BrowserMatch ^-?$ keep_out
BrowserMatch Ahrefs keep_out
BrowserMatch "America Online Browser" keep_out
BrowserMatch Clipish keep_out
BrowserMatch Covario keep_out
BrowserMatch CoverScout keep_out
BrowserMatch "Extreme Picture Finder" keep_out
BrowserMatch FairShare keep_out
BrowserMatch HTTrack keep_out
BrowserMatch "Jakarta Commons-HttpClient/3.1" keep_out
BrowserMatch "Java/" keep_out
BrowserMatch libcurl keep_out
BrowserMatch "MSIE [1-4]\." keep_out
BrowserMatch "Mozilla/[0-3]" keep_out
BrowserMatch NativeHost keep_out
BrowserMatch Python keep_out
BrowserMatch scanner keep_out
BrowserMatch TencentTraveler keep_out
BrowserMatch vcbot keep_out
BrowserMatch webcollage keep_out
BrowserMatch Wget keep_out
BrowserMatch Wikimpress keep_out
# comment-out following for link checker
BrowserMatch libwww-perl keep_out
BrowserMatch Yahoo keep_out


Goes without saying that I ran the linkchecker a few hours ago-- and forgot to comment-out that libwww-perl line. Luckily I was checking locally so only one link was affected.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4521718.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com