Welcome to WebmasterWorld Guest from 54.159.44.227

Forum Moderators: open

Message Too Old, No Replies

slurped to death

     

oilman

10:38 pm on Sep 18, 2000 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



ok - I got slurped 3 times today from 3 different IPs. Those didn't bother me - they were all known but then this showed up and only grabbed my main page:

ip: 209.185.208.29
reverse DNS: ALLISONB
from: Exodus Communications
Time: 6 min
Browser: Mozilla/4.0 (compatible; MSIE 5.01; Window NT 5.0)
refering url: www.some.real.url

Is this someone from Exodus legitimatly checking out our product by following the url from somewhere or is this an arachnoid that I don't have in my list?

Also saw this little guy: 209.185.240.250 - didn't have time to dig into it yet but I know it's not on my list.

Kamikaze

1:23 am on Sep 19, 2000 (gmt 0)

10+ Year Member



Oilman- Would you happen to know if it grabbed your robots.txt file?

oilman

1:32 am on Sep 19, 2000 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The first one sure didn't. I was pretty disappointed that it didn't. ;) That makes life so much easier.

Can't say for sure on that second IP though.

Gorufu

12:43 pm on Sep 21, 2000 (gmt 0)

10+ Year Member



Hi oilman,

209.185.208.29 appears to be a legitimate visitor, especially if there was a referring URL.

209.185.240.250 www.law3.hotmail.com

BTW: I did a little bit of digging and found that
Exodus Communications is the NetBlock maintainer for
directhit.com
hotmail.com
hotbot.com
lycos.com
iwon.com
alltheweb.com
Inktomi (most Slurp spidering servers)
inktomi.com (NS3.INKTOMI.COM NS4.INKTOMI.COM)

Doing a traceroute all of the above domains go through
exodus-bos1-oc3.customer.ALTER.NET (157.130.15.202)
then different exodus.net servers before their final destination

oilman

3:13 pm on Sep 21, 2000 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks Gorufu - that's what I figured as well. Welcome to the forums.

On another note: Anybody know what the y400 spider does for INK? She has been hitting me regularly for the past week at least 3 times a day. Pulls robots.txt and default.asp. Take about 6 minutes to do it then leaves.

oilman

7:58 pm on Sep 22, 2000 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Here's my log for y400 for the last 2 days:
anybody else seeing this kind of activity?

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-20 09:30:37 - user created
2000-09-20 09:30:37 /robots.txt
2000-09-20 09:30:40 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-20 15:31:21 - user created
2000-09-20 15:31:21 /robots.txt
2000-09-20 15:31:22 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 01:33:48 - user created
2000-09-21 01:33:48 /robots.txt
2000-09-21 01:33:58 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 11:21:38 - user created
2000-09-21 11:21:38 /robots.txt
2000-09-21 11:21:51 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 13:31:30 - user created
2000-09-21 13:31:30 /robots.txt
2000-09-21 13:31:31 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 15:33:14 - user created
2000-09-21 15:33:14 /robots.txt
2000-09-21 15:33:19 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 16:40:25 - user created
2000-09-21 16:40:25 /robots.txt
2000-09-21 16:40:25 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 21:29:22 - user created
2000-09-21 21:29:22 /robots.txt
2000-09-21 21:29:23 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-22 03:29:06 - user created
2000-09-22 03:29:06 /robots.txt
2000-09-22 03:29:12 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-22 09:29:58 - user created
2000-09-22 09:29:58 /robots.txt
2000-09-22 09:30:00 /default.asp

drbill

1:15 pm on Sep 25, 2000 (gmt 0)

10+ Year Member



Oilman,

Slurp has been living on my server for the last 3 weeks.

Hitting hard but I see nothing in the way of traffic from any of the Ink Portals.

Brett_Tabke

1:34 pm on Sep 25, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Gorufu, thank you - that explains alot. I didn't know just what it was Exodus did, but knew many spiders sure ran through their system. Not a whole lot about them on the net.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month