slurped to death

[oilman]

ok - I got slurped 3 times today from 3 different IPs. Those didn't bother me - they were all known but then this showed up and only grabbed my main page:

ip: 209.185.208.29
reverse DNS: ALLISONB
from: Exodus Communications
Time: 6 min
Browser: Mozilla/4.0 (compatible; MSIE 5.01; Window NT 5.0)
refering url: www.some.real.url

Is this someone from Exodus legitimatly checking out our product by following the url from somewhere or is this an arachnoid that I don't have in my list?

Also saw this little guy: 209.185.240.250 - didn't have time to dig into it yet but I know it's not on my list.

[Kamikaze]

Oilman- Would you happen to know if it grabbed your robots.txt file?

[oilman]

The first one sure didn't. I was pretty disappointed that it didn't. ;) That makes life so much easier.

Can't say for sure on that second IP though.

[Gorufu]

Hi oilman,

209.185.208.29 appears to be a legitimate visitor, especially if there was a referring URL.

209.185.240.250 www.law3.hotmail.com

BTW: I did a little bit of digging and found that
Exodus Communications is the NetBlock maintainer for
directhit.com
hotmail.com
hotbot.com
lycos.com
iwon.com
alltheweb.com
Inktomi (most Slurp spidering servers)
inktomi.com (NS3.INKTOMI.COM NS4.INKTOMI.COM)

Doing a traceroute all of the above domains go through
exodus-bos1-oc3.customer.ALTER.NET (157.130.15.202)
then different exodus.net servers before their final destination

[oilman]

Thanks Gorufu - that's what I figured as well. Welcome to the forums.

On another note: Anybody know what the y400 spider does for INK? She has been hitting me regularly for the past week at least 3 times a day. Pulls robots.txt and default.asp. Take about 6 minutes to do it then leaves.

[oilman]

Here's my log for y400 for the last 2 days:
anybody else seeing this kind of activity?

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-20 09:30:37 - user created
2000-09-20 09:30:37 /robots.txt
2000-09-20 09:30:40 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-20 15:31:21 - user created
2000-09-20 15:31:21 /robots.txt
2000-09-20 15:31:22 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 01:33:48 - user created
2000-09-21 01:33:48 /robots.txt
2000-09-21 01:33:58 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 11:21:38 - user created
2000-09-21 11:21:38 /robots.txt
2000-09-21 11:21:51 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 13:31:30 - user created
2000-09-21 13:31:30 /robots.txt
2000-09-21 13:31:31 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 15:33:14 - user created
2000-09-21 15:33:14 /robots.txt
2000-09-21 15:33:19 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 16:40:25 - user created
2000-09-21 16:40:25 /robots.txt
2000-09-21 16:40:25 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-21 21:29:22 - user created
2000-09-21 21:29:22 /robots.txt
2000-09-21 21:29:23 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-22 03:29:06 - user created
2000-09-22 03:29:06 /robots.txt
2000-09-22 03:29:12 /default.asp

* User Session 209.185.141.226 - Exodus Commnications Inc. (CA,US)
2000-09-22 09:29:58 - user created
2000-09-22 09:29:58 /robots.txt
2000-09-22 09:30:00 /default.asp

[drbill]

Oilman,

Slurp has been living on my server for the last 3 weeks.

Hitting hard but I see nothing in the way of traffic from any of the Ink Portals.

[Brett_Tabke]

Gorufu, thank you - that explains alot. I didn't know just what it was Exodus did, but knew many spiders sure ran through their system. Not a whole lot about them on the net.