homepage Welcome to WebmasterWorld Guest from 54.211.7.174
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
possible spider...
216.161.154.93 ua Mozilla/4.0+(compatible;+MSIE+5.0;+Windows
han solo




msg:401793
 3:59 pm on Nov 29, 2000 (gmt 0)

216.161.154.93 Mozilla/4.0+(compatible;+MSIE+5.0;+Windows+NT;+DigExt)

Crawled some of my stuff, I thought it was a surfer, but I didn't see it come from any engine, and the amount of stuff it grabbed really through me for a loop, especially the regularity of the requessts.

Does any one know if this is a spider, or if some surfer just couldn't get enough of my redirect system, and had to click on everything?

Thanks,
han solo

 

BoneHeadicus




msg:401794
 4:22 pm on Nov 29, 2000 (gmt 0)


216.161.154.93
Host unreachable

216.160.0.0 - 216.161.255.255

U S WEST Interact Services
600 Stinson Blvd NE
Minneapolis, MN 55413
us

U S WEST ISOps abuse@uswest.net
612-664-4689

NS1.USWEST.NET 204.147.80.5
NS2.DNVR.USWEST.NET 206.196.128.1

USW-INTERACT98
Updated: 18-Jul-2000
Source: whois.arin.net

Of course you already knew this.

Does it usually mean when the host is unreachable that it most likely is a dial-up account?

eljefe3




msg:401795
 4:32 pm on Nov 29, 2000 (gmt 0)

I have a program (teleport pro) that I can run from my desktop which spiders an entire site and doesn't leave a referrer if you want to remain anonymous. It will follow links up to 3 deep. It will leave your dial-up ip address though. Perhaps this guy is using the same.

han solo




msg:401796
 4:35 pm on Nov 29, 2000 (gmt 0)

Dial up accounts, and dsl accounts, I believe look a little different, but then, I'm certainly not an expert.

The reason I posted this was the activity on my logs...it seemed more like a spider than a surfer, and I handed it the information I give to web surfers, and not to spiders...

What I've been doing is custom serving different entities, and since this "user" didn't seem to come from a search engine, I got confused, since it seemed like a surfer with the ip/ua combo it gave...but then when it grabbed so much stuff...makes me wonder.

Perhaps the gods of seo will be along shortly to help us...;)

Cheers,
Han Solo

eljefe3




msg:401797
 4:12 am on Nov 30, 2000 (gmt 0)

han solo,

What I was trying to say here is that there is a program which will come to your site and look like a spider. It gives no UA (if you don't want it to) and no referrer. The only way to determine whether it is a search engine or a surfer running his own bot is by ip. As long as you are cloaking by ip delivery, you are OK.

eljefe3




msg:401798
 4:13 am on Nov 30, 2000 (gmt 0)

double post

Edited by: eljefe3

han solo




msg:401799
 2:50 pm on Nov 30, 2000 (gmt 0)

Thanks. Teleport pro? I'll have to make sure I follow up on that. I've been filtering by all I can, usually both, sometimes one, sometimes the other, depending on how easy the data appears...

With some of the engines, surfers doing odd things, I've begun to get very paranoid...after a while, if anything comes through and looks even slightly abnormal, I spend at least an hour tracking it down, checking the ip, and trying to assure myself I didn't just redirect the latest version of slurp :)

Thanks again for your help.

Cheers,
Han Solo

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved