I have not seen this one before.

Forum Moderators: open

Message Too Old, No Replies

I have not seen this one before.

miles

4:41 pm on Apr 13, 2001 (gmt 0)

Does anyone know who this spider belongs to 209.247.40.106

Thanks.

theperlyking

4:43 pm on Apr 13, 2001 (gmt 0)

Shows as an Alexa IP , see [alexa.com...]

I've had this crawl me a few times, rarely get any people referred from it though.

miles

4:50 pm on Apr 13, 2001 (gmt 0)

Thanks for the info.

BoneHeadicus

4:54 pm on Apr 13, 2001 (gmt 0)

Standard issue robots.txt:

User-agent: ia_archiver
Disallow: /

Alexa and I broke up a while back cuz she doesn't produce. She wanted to keep tabs on me but there's No Such Agreement on my part.

msgraph

5:08 pm on Apr 13, 2001 (gmt 0)

How do they find our sites? Do they pull them off a specific search engine?Or do they just follow links from wherever they find them to an endless abyss?

I'm thinking about banning them too BH, but I want to know if they are selling their database out to any of the major players. Or at least set a limit to how many files they can grab. They are eating up way too much bandwidth

theperlyking

5:13 pm on Apr 13, 2001 (gmt 0)

Arent alexa the ones who "power" the whats related feature in netscape (and maybe IE). If so perhaps this is where they pick our sites up from.
edit: - yep see [home.netscape.com...]