homepage Welcome to WebmasterWorld Guest from 184.72.82.126
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Just a few more
who/what/how
luckynh




msg:399752
 3:10 pm on Mar 19, 2001 (gmt 0)

Any takers on who/what/how to Disallow them from grabbing my pages?

IP: 204.167.146.68
UA: HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+4.0+Robot)+Microsoft

IP 195.180.245.163
UA HTTP/1.0 Spinne/2.0

IP 213.46.124.11
UA HTTP/1.0 vspider

IP 128.93.11.67
UA HTTP/1.1 cosmos/0.5

IP 198.4.83.49
UA HTTP/1.0 WebFountain

IP 193.251.190.3
UA HTTP/1.0 EmailSiphon WEBTRENDS_ID=193.251.190.3-1992041584.29404971;+expires=Fri,+31-Dec-2010+00:00:00+GMT;+path=/

IP 64.32.12.115
UA HTTP/1.0 icdpos/1.0

IP 64.28.94.13
UA HTTP/1.0 beholder+(www.vigiltech.com/esensedisclaim.html)

IP XXX.XXX.XXX.XXX (dont think i should post thier ip)
UA Zeus+81212+Webster+Pro+V2
I know what Zeus is but does it obey a robots.txt file? better yet how can i make sure that it does not index pages that are not for the general public? (i.e cgi-bin...ect.)

 

littleman




msg:399753
 7:38 am on Mar 20, 2001 (gmt 0)

Ok, I'll start.
EmailSiphon - an email thief.

IP 213.46.124.11
UA HTTP/1.0 vspider
d124011.upc-d.chello.nl
I believe Chello is an Inktomi partner.

Rumbas




msg:399754
 9:04 am on Mar 20, 2001 (gmt 0)

I've been hit on every domain with this one too:

IP 195.180.245.163
UA HTTP/1.0 Spinne/2.0

I think it is a german spider, but not sure from who?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved