Welcome to WebmasterWorld Guest from 50.19.53.104

Forum Moderators: goodroi

Message Too Old, No Replies

MSIECrawler not obeying my robots.txt :(

any ideas ?

     

diddlydazz

2:36 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



in my robots.txt i have :

User-Agent:MSIECrawler
Disallow:/

Any ideas why it doesn't obey ?

Any help greatly appreciated

Dazz

Brett_Tabke

2:40 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



What's the ip?

diddlydazz

2:42 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



32.97.239.35

It traces to Bedford, NH

I presume this is someone making it available offline Brett ?

Brett_Tabke

3:02 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Just making sure it wasn't an se running it.

Ya, it was someone just running a spider to scrap your site. Most of those don't even read the robots.txt, let alone obey it.

diddlydazz

3:09 am on Jan 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks Brett, I should just let them get on with it then :)

Thanks

Dazz

 

Featured Threads

Hot Threads This Week

Hot Threads This Month