Welcome to WebmasterWorld Guest from 54.196.243.192

Forum Moderators: goodroi

Message Too Old, No Replies

MSIECrawler not obeying my robots.txt :(

any ideas ?

     
2:36 am on Jan 12, 2002 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Nov 27, 2001
posts:1156
votes: 2


in my robots.txt i have :

User-Agent:MSIECrawler
Disallow:/

Any ideas why it doesn't obey ?

Any help greatly appreciated

Dazz

2:40 am on Jan 12, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


What's the ip?
2:42 am on Jan 12, 2002 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Nov 27, 2001
posts:1156
votes: 2


32.97.239.35

It traces to Bedford, NH

I presume this is someone making it available offline Brett ?

3:02 am on Jan 12, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


Just making sure it wasn't an se running it.

Ya, it was someone just running a spider to scrap your site. Most of those don't even read the robots.txt, let alone obey it.

3:09 am on Jan 12, 2002 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Nov 27, 2001
posts:1156
votes: 2


Thanks Brett, I should just let them get on with it then :)

Thanks

Dazz