Welcome to WebmasterWorld Guest from 18.104.22.168 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor 2014
Why would rogue spiders obey robots.txt? irock msg:1528495 11:44 pm on Jan 8, 2003 (gmt 0) Hmm...
I just checked out Webmasterworld's robots.txt, and it would seem there are plenty of rogue spiders trying to duplicate sites or whatever... so, i was wondering if these spiders are just sent out by unethical people, why would they obey the rules in robots.txt.
EDIT: Oops... I posted it on the wrong forum...
hakre msg:1528496 5:29 am on Jan 9, 2003 (gmt 0)
yes that's true. on the other hand it would be a real news if google would do... :)