homepage Welcome to WebmasterWorld Guest from 54.197.147.90
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
WebmasterWorld disallowing Mozilla, Why?
If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that
bufferzone




msg:1528354
 1:22 pm on Mar 14, 2004 (gmt 0)

Take a look at Brett's Robots.txt at [webmasterworld.com...] . In it Brett disallows Mozilla in all it's versions. Why is that? An internet user using Mozilla will not look at robots.txt and even if he does, hi will probably not obey it anyway.

Can anyone explain, please?

Regards
Kim

 

choster




msg:1528355
 8:23 pm on Mar 15, 2004 (gmt 0)

They're disallowing automated spiders/crawlers, not browsers. Some spiders/crawlers spoof their user agent string, pretending to be a browser, to get around such filters; it would seem Brett is trying to head them off. A human browsing the site, as you noted, will not be affected by robots.txt and will be able to access all the content.

bufferzone




msg:1528356
 8:54 pm on Mar 15, 2004 (gmt 0)

Thanks

Quit obvius now that you have pointe it out in plain words

g1smd




msg:1528357
 10:47 pm on Mar 31, 2004 (gmt 0)

I use the Mozilla browser. I can read the site and post in threads.

Q.E.D.

Brett_Tabke




msg:1528358
 10:59 pm on Mar 31, 2004 (gmt 0)

choster has it. Some bots allow people to change the agent name. Often, that bot will still check a robots.txt with the new name. So, by putting moz/ie in the robots.txt we deny those that are smart enough to change the agent name, but not the bot behavior to check the bots.txt

g1smd, browsers don't check robots.txt - only bots that adherd to the robots exclusion proposal.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved