Welcome to WebmasterWorld Guest from 18.104.22.168 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
I have not seen this one before. miles
Does anyone know who this spider belongs to 22.214.171.124
Shows as an Alexa IP , see [ ...] alexa.com
I've had this crawl me a few times, rarely get any people referred from it though.
Thanks for the info. BoneHeadicus
Standard issue robots.txt:
Alexa and I broke up a while back cuz she doesn't produce. She wanted to keep tabs on me but there's No Such Agreement on my part.
How do they find our sites? Do they pull them off a specific search engine?Or do they just follow links from wherever they find them to an endless abyss?
I'm thinking about banning them too BH, but I want to know if they are selling their database out to any of the major players. Or at least set a limit to how many files they can grab. They are eating up way too much bandwidth
Arent alexa the ones who "power" the whats related feature in netscape (and maybe IE). If so perhaps this is where they pick our sites up from. edit: - yep see [ ...] home.netscape.com