Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Automatically detect crawlers / bots


Globetrotter - 9:43 pm on Dec 7, 2011 (gmt 0)


No wonder I couldn't get my version to work, it seems it was missing huge important parts. So Thanks for the link to the full version.

I guess it would be smart to add some kind of feedback per mail when some bot is caught?

And what kind of bots do you allow? E.g. the social media report sites for companies when you or a user mention their name? And what crawler is used for Google alerts?


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4395123.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com