Welcome to WebmasterWorld Guest from 220.127.116.11 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
I have not seen this one before. miles msg:399680 4:41 pm on Apr 13, 2001 (gmt 0) Does anyone know who this spider belongs to 18.104.22.168
theperlyking msg:399681 4:43 pm on Apr 13, 2001 (gmt 0)
Shows as an Alexa IP , see [ ...] alexa.com
I've had this crawl me a few times, rarely get any people referred from it though.
miles msg:399682 4:50 pm on Apr 13, 2001 (gmt 0)
Thanks for the info. BoneHeadicus msg:399683 4:54 pm on Apr 13, 2001 (gmt 0)
Standard issue robots.txt:
Alexa and I broke up a while back cuz she doesn't produce. She wanted to keep tabs on me but there's No Such Agreement on my part.
msgraph msg:399684 5:08 pm on Apr 13, 2001 (gmt 0)
How do they find our sites? Do they pull them off a specific search engine?Or do they just follow links from wherever they find them to an endless abyss?
I'm thinking about banning them too BH, but I want to know if they are selling their database out to any of the major players. Or at least set a limit to how many files they can grab. They are eating up way too much bandwidth
theperlyking msg:399685 5:13 pm on Apr 13, 2001 (gmt 0)
Arent alexa the ones who "power" the whats related feature in netscape (and maybe IE). If so perhaps this is where they pick our sites up from. edit: - yep see [ ...] home.netscape.com