Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

possible spider... ua Mozilla/4.0+(compatible;+MSIE+5.0;+Windows


han solo

3:59 pm on Nov 29, 2000 (gmt 0) Mozilla/4.0+(compatible;+MSIE+5.0;+Windows+NT;+DigExt)

Crawled some of my stuff, I thought it was a surfer, but I didn't see it come from any engine, and the amount of stuff it grabbed really through me for a loop, especially the regularity of the requessts.

Does any one know if this is a spider, or if some surfer just couldn't get enough of my redirect system, and had to click on everything?

han solo


4:22 pm on Nov 29, 2000 (gmt 0)

10+ Year Member
Host unreachable -

U S WEST Interact Services
600 Stinson Blvd NE
Minneapolis, MN 55413

U S WEST ISOps abuse@uswest.net


Updated: 18-Jul-2000
Source: whois.arin.net

Of course you already knew this.

Does it usually mean when the host is unreachable that it most likely is a dial-up account?


4:32 pm on Nov 29, 2000 (gmt 0)

WebmasterWorld Administrator 10+ Year Member

I have a program (teleport pro) that I can run from my desktop which spiders an entire site and doesn't leave a referrer if you want to remain anonymous. It will follow links up to 3 deep. It will leave your dial-up ip address though. Perhaps this guy is using the same.

han solo

4:35 pm on Nov 29, 2000 (gmt 0)

Dial up accounts, and dsl accounts, I believe look a little different, but then, I'm certainly not an expert.

The reason I posted this was the activity on my logs...it seemed more like a spider than a surfer, and I handed it the information I give to web surfers, and not to spiders...

What I've been doing is custom serving different entities, and since this "user" didn't seem to come from a search engine, I got confused, since it seemed like a surfer with the ip/ua combo it gave...but then when it grabbed so much stuff...makes me wonder.

Perhaps the gods of seo will be along shortly to help us...;)

Han Solo


4:12 am on Nov 30, 2000 (gmt 0)

WebmasterWorld Administrator 10+ Year Member

han solo,

What I was trying to say here is that there is a program which will come to your site and look like a spider. It gives no UA (if you don't want it to) and no referrer. The only way to determine whether it is a search engine or a surfer running his own bot is by ip. As long as you are cloaking by ip delivery, you are OK.


4:13 am on Nov 30, 2000 (gmt 0)

WebmasterWorld Administrator 10+ Year Member

double post

Edited by: eljefe3

han solo

2:50 pm on Nov 30, 2000 (gmt 0)

Thanks. Teleport pro? I'll have to make sure I follow up on that. I've been filtering by all I can, usually both, sometimes one, sometimes the other, depending on how easy the data appears...

With some of the engines, surfers doing odd things, I've begun to get very paranoid...after a while, if anything comes through and looks even slightly abnormal, I spend at least an hour tracking it down, checking the ip, and trying to assure myself I didn't just redirect the latest version of slurp :)

Thanks again for your help.

Han Solo


Featured Threads

Hot Threads This Week

Hot Threads This Month