homepage Welcome to WebmasterWorld Guest from 54.163.89.8
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Whois.sc bot goes undercover
No way to opt out?
fiestagirl

10+ Year Member



 
Msg#: 1665 posted 6:32 am on Feb 12, 2003 (gmt 0)

I know that this has been discussed before.
[webmasterworld.com...]

But -Now they are back to spidering without identifying themselves. They have stated on their info page that their bot does not read robots.txt and that there is no way to opt out of being probed by them. So for those of us that would like to "opt out".

We see two different user agents for these visits. Sometimes, the bot shows us
user agent = "SurveyBot/2.2" followed by a cheap link to the site complete with anchor text (just in case your log files getlisted in GG). Other times the bot attempts to pretend to be a browser with this user agent:
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"

The referrer will be "http://www(link to the site in question).sc" or "http://www.virtual-private-server.com/"

This bot uses these ip addresses:

216.145.11.94
38.116.0.122
198.78.175.190
198.78.175.42
38.116.0.6
66.228.210.10
66.228.210.50
66.228.202.22

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 1665 posted 11:19 am on Feb 12, 2003 (gmt 0)

Ever notice GoogleBot following that bot? (just wondering).

volatilegx

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1665 posted 4:36 pm on Feb 12, 2003 (gmt 0)

Do you have any evidence this bot and Googlebot might be related, Brett?

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 1665 posted 4:41 pm on Feb 12, 2003 (gmt 0)

no. just asking the question is all. (fishing expedition based on several fish in the boat).

fiestagirl

10+ Year Member



 
Msg#: 1665 posted 4:51 pm on Feb 12, 2003 (gmt 0)

No I haven't noticed that. Of course I have had them banned since before they began to identify themselves. Call me stubborn but I refuse to play the game with any copyright bot, plagarism bot, server survey bot, etc.

We reserve the right to refuse service to anyone - as it were.

What a great way to start a rumor and get the ban lifted by some people out of fear though.

Finder

10+ Year Member



 
Msg#: 1665 posted 9:40 pm on Feb 20, 2003 (gmt 0)

You can add 216.145.5.42 to the list.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved