homepage Welcome to WebmasterWorld Guest from 54.226.180.223
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Alltheweb spider
stn24




msg:406742
 4:58 pm on Mar 10, 2001 (gmt 0)

I just got hit by: 209.67.247.232 - user agent: Mozilla/4.0 (compatible; MSIE 4.01; Win32).

DNS resolves to : 209.67.247.232 Host: rdcrawler3.uswal.alltheweb.com
Alternate address: 50.51.50.46
Alternate address: 50.52.55.46
Alternate address: 54.55.46.50
Alternate address: 48.57.46.105
Alternate address: 110.45.97.100
Alternate address: 100.114.46.97
Aliases: 232.247.67.209.in-addr.arpa

Thank god for straight ip cloaking

I looked through my files and this is what the UA used to look like: rdcrawler3.uswal.alltheweb.com

 

BoneHeadicus




msg:406743
 5:13 pm on Mar 10, 2001 (gmt 0)

Yes yes. They are getting sneaky.

I would suggest keeping a very close eye on Fast....there might be more like that.

mivox




msg:406744
 12:29 am on Mar 11, 2001 (gmt 0)

in-addr.arpa? Is there a specific IP range of in-addr.arpa that's spiders, or are they ALL spiders... I get regular hits from in-addr.arpa machines, and I could never figure them out.

mivox




msg:406745
 12:29 am on Mar 11, 2001 (gmt 0)

Gosh! My first duplicate post!

stn24




msg:406746
 5:56 am on Mar 11, 2001 (gmt 0)

I really don't know, here is what I've got :
208.51.0.79
209.67.247.204
209.67.247.203
209.67.247.201
209.67.247.157
209.67.247.156
209.67.247.155
209.202.148.41
209.202.148.36
209.202.148.35
All these resolve to in-addr.arpa

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved