Welcome to WebmasterWorld Guest from 54.166.48.3

Forum Moderators: goodroi

Message Too Old, No Replies

WebmasterWorld disallowing Mozilla, Why?

If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that

     
1:22 pm on Mar 14, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 24, 2001
posts:562
votes: 0


Take a look at Brett's Robots.txt at [webmasterworld.com...] . In it Brett disallows Mozilla in all it's versions. Why is that? An internet user using Mozilla will not look at robots.txt and even if he does, hi will probably not obey it anyway.

Can anyone explain, please?

Regards
Kim

8:23 pm on Mar 15, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 17, 2001
posts:1262
votes: 0


They're disallowing automated spiders/crawlers, not browsers. Some spiders/crawlers spoof their user agent string, pretending to be a browser, to get around such filters; it would seem Brett is trying to head them off. A human browsing the site, as you noted, will not be affected by robots.txt and will be able to access all the content.
8:54 pm on Mar 15, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 24, 2001
posts:562
votes: 0


Thanks

Quit obvius now that you have pointe it out in plain words

10:47 pm on Mar 31, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


I use the Mozilla browser. I can read the site and post in threads.

Q.E.D.

10:59 pm on Mar 31, 2004 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38066
votes: 15


choster has it. Some bots allow people to change the agent name. Often, that bot will still check a robots.txt with the new name. So, by putting moz/ie in the robots.txt we deny those that are smart enough to change the agent name, but not the bot behavior to check the bots.txt

g1smd, browsers don't check robots.txt - only bots that adherd to the robots exclusion proposal.