Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Secure Sites From Botnet Vulnerability Probes


amznVibe - 11:04 pm on May 11, 2007 (gmt 0)


Just a caution about false triggers:

Some legit popular tools like flvplayer use urls like?file=http://example.com/

I suspect some places that pickup your RSS feeds, which you might have no problem with if they send you traffic, will have the default user agent still in place of "libwww-perl"


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3335216.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com