Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: lawman
How about changing:
"Search Engine Spider Identification"
"Search Engine & Crawler Identification"
joined:Apr 25, 2002
Might not be a bad idea to lump the good guys with the bad guys in one set of discussions since IDing the good guys is only an issue because there are bad guys and vice versa.
With the charter being to help identify automated activity from Search Engines, Crawlers, Link Checkers, Scrapers, Botnets, etc."
Just a thought.
JD is on the right track there, but I wonder how we avoid the privacy issues?
We don't like to get into senarios where we are id'ing private individuals
And I think that's why Brett you guys made that forum pre-moderated, also hiding the last IP octet rule except for recognized search engines takes care of those scenarios.
If anything, we need to spot, analyze and discuss everything new crawling our sites search engines or not, specially those originating from known hosting data centers, this interest ranges from business survival down to a hobby levels like plane & train spotting, I sure like to know more about them before getting run over by one.
Deep inside I wish it becomes as simple as "Crawler Identification", search engines thrown in as a bonus :-)