homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Whois.sc bot goes undercover
No way to opt out?

 6:32 am on Feb 12, 2003 (gmt 0)

I know that this has been discussed before.

But -Now they are back to spidering without identifying themselves. They have stated on their info page that their bot does not read robots.txt and that there is no way to opt out of being probed by them. So for those of us that would like to "opt out".

We see two different user agents for these visits. Sometimes, the bot shows us
user agent = "SurveyBot/2.2" followed by a cheap link to the site complete with anchor text (just in case your log files getlisted in GG). Other times the bot attempts to pretend to be a browser with this user agent:
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"

The referrer will be "http://www(link to the site in question).sc" or "http://www.virtual-private-server.com/"

This bot uses these ip addresses:



 11:19 am on Feb 12, 2003 (gmt 0)

Ever notice GoogleBot following that bot? (just wondering).


 4:36 pm on Feb 12, 2003 (gmt 0)

Do you have any evidence this bot and Googlebot might be related, Brett?


 4:41 pm on Feb 12, 2003 (gmt 0)

no. just asking the question is all. (fishing expedition based on several fish in the boat).


 4:51 pm on Feb 12, 2003 (gmt 0)

No I haven't noticed that. Of course I have had them banned since before they began to identify themselves. Call me stubborn but I refuse to play the game with any copyright bot, plagarism bot, server survey bot, etc.

We reserve the right to refuse service to anyone - as it were.

What a great way to start a rumor and get the ban lifted by some people out of fear though.


 9:40 pm on Feb 20, 2003 (gmt 0)

You can add to the list.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved