Welcome to WebmasterWorld Guest from 54.163.49.19

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

I have not seen this one before.

     
4:41 pm on Apr 13, 2001 (gmt 0)

Full Member

10+ Year Member

joined:Dec 12, 2000
posts:237
votes: 0


Does anyone know who this spider belongs to 209.247.40.106

Thanks.

4:43 pm on Apr 13, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 21, 2001
posts:419
votes: 0


Shows as an Alexa IP , see [alexa.com...]

I've had this crawl me a few times, rarely get any people referred from it though.

4:50 pm on Apr 13, 2001 (gmt 0)

Full Member

10+ Year Member

joined:Dec 12, 2000
posts:237
votes: 0


Thanks for the info.
4:54 pm on Apr 13, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 25, 2000
posts:541
votes: 0


Standard issue robots.txt:

User-agent: ia_archiver
Disallow: /

Alexa and I broke up a while back cuz she doesn't produce. She wanted to keep tabs on me but there's No Such Agreement on my part.

5:08 pm on Apr 13, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 29, 2000
posts:1425
votes: 0


How do they find our sites? Do they pull them off a specific search engine?Or do they just follow links from wherever they find them to an endless abyss?

I'm thinking about banning them too BH, but I want to know if they are selling their database out to any of the major players. Or at least set a limit to how many files they can grab. They are eating up way too much bandwidth

5:13 pm on Apr 13, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 21, 2001
posts:419
votes: 0


Arent alexa the ones who "power" the whats related feature in netscape (and maybe IE). If so perhaps this is where they pick our sites up from.
edit: - yep see [home.netscape.com...]