Welcome to WebmasterWorld Guest from 54.197.116.116

Forum Moderators: goodroi

Message Too Old, No Replies

MSIECrawler not obeying my robots.txt :(

any ideas ?

   
2:36 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



in my robots.txt i have :

User-Agent:MSIECrawler
Disallow:/

Any ideas why it doesn't obey ?

Any help greatly appreciated

Dazz

2:40 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



What's the ip?
2:42 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



32.97.239.35

It traces to Bedford, NH

I presume this is someone making it available offline Brett ?

3:02 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Just making sure it wasn't an se running it.

Ya, it was someone just running a spider to scrap your site. Most of those don't even read the robots.txt, let alone obey it.

3:09 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks Brett, I should just let them get on with it then :)

Thanks

Dazz