Welcome to WebmasterWorld Guest from 184.108.40.206
Any ideas why it doesn't obey ?
Any help greatly appreciated
It traces to Bedford, NH
I presume this is someone making it available offline Brett ?
Ya, it was someone just running a spider to scrap your site. Most of those don't even read the robots.txt, let alone obey it.
Become a Pro Member