Welcome to WebmasterWorld Guest from 22.214.171.124
joined:Nov 27, 2001
Any ideas why it doesn't obey ?
Any help greatly appreciated
joined:Sept 21, 1999
It traces to Bedford, NH
I presume this is someone making it available offline Brett ?
Ya, it was someone just running a spider to scrap your site. Most of those don't even read the robots.txt, let alone obey it.
Become a Pro Member