Spying spiders - do you know (m)any of them

[smallcompany]

I just made a post in AdWords thread about seeing (and banning from now on) SpyFu related spider.

Now, when I better think, I'm sure there are tons of similar tools out there that "spy" on our sites, AdWords campaigns, etc.

So I wonder if anyone is maintaining a list of those that are not to be allowed access to our sites.

Thanks

[keyplyr]

The problem with any "list" is that it will differ for each webmaster. What's considered a threat to some may be a benefit to others, so each person really needs to do their own specific research and keep a diligent eye on their server access logs.

That said, this Search Engine Spider and User Agent Identification forum has archived a vast amount of info on various UAs and IP ranges that have drawn attention to themselves for one reason or another.

Most of us use a combination of blocking by IP address and/or UA strings in conjunction with allowing access through whitelists.

Thanks for the heads-up regarding SpyFu. What's the complete UA and IP range?

[blend27]

Most of those, "EXTRA NICE TOOLZ", try to use some type of Screen-Shot addee, I have a 'spy-all-you-want' domain that I use as a 'pet', just running the query for that domain most likely will result in them visiting it when the query is run. And then 403 on the rest of the sites.

Most of those "SPY" sites are targetting a novice webmaster on "We have Keyword Data" type of way that did not get the memo just yet, but in reality one could get much better answer to the question on 'YoWikiHooBundleGalaBassGuDotBam Answers'.

And then there is a user that typed your domain name in searchengines search box, if the 'SPY' company comes up on the first page they have a chance of a user visiting their site and maybe even ckicking on an Add or 2.

[Pfui]

@ keyplyr: No specific Host info, but I've seen SpyFu as a referer and long ago blocked same. Example formats (circa 2008):

http://www.spyf*.com/Term.aspx?t=****&org=**
http://www.spyf*.com/Term.aspx?t=*****

[keyplyr]

Thanks Pfui