Welcome to WebmasterWorld Guest from 54.144.243.34

Forum Moderators: Ocean10000 & incrediBILL

I have not seen this one before.

   
4:41 pm on Apr 13, 2001 (gmt 0)

10+ Year Member



Does anyone know who this spider belongs to 209.247.40.106

Thanks.

4:43 pm on Apr 13, 2001 (gmt 0)

10+ Year Member



Shows as an Alexa IP , see [alexa.com...]

I've had this crawl me a few times, rarely get any people referred from it though.

4:50 pm on Apr 13, 2001 (gmt 0)

10+ Year Member



Thanks for the info.
4:54 pm on Apr 13, 2001 (gmt 0)

10+ Year Member



Standard issue robots.txt:

User-agent: ia_archiver
Disallow: /

Alexa and I broke up a while back cuz she doesn't produce. She wanted to keep tabs on me but there's No Such Agreement on my part.

5:08 pm on Apr 13, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



How do they find our sites? Do they pull them off a specific search engine?Or do they just follow links from wherever they find them to an endless abyss?

I'm thinking about banning them too BH, but I want to know if they are selling their database out to any of the major players. Or at least set a limit to how many files they can grab. They are eating up way too much bandwidth

5:13 pm on Apr 13, 2001 (gmt 0)

10+ Year Member



Arent alexa the ones who "power" the whats related feature in netscape (and maybe IE). If so perhaps this is where they pick our sites up from.
edit: - yep see [home.netscape.com...]
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month