homepage Welcome to WebmasterWorld Guest from 54.234.74.85
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
mozDex is spidering
FYI on the user agent
ByronM




msg:1528533
 9:12 pm on Apr 1, 2004 (gmt 0)

Our bot follows the robots.txt exclusion standard, which is described at [robotstxt.org...] Depending on the configuration, our robot may obey different rules. To make it simple to send our bot away, we'll always obey rules for "mozDex". Here are the different cases.
When we're running to populate our index, we'll advertise the agent "mozDex", and obey rules for "mozDex" if they exist, or "mozdex", or "*".

To ban all bots from your site, place the following in your robots.txt file:

User-agent: *

Disallow: /

To ban mozDex bots from your site, place the following in your robots.txt file:

User-agent: mozDex

Disallow: /

Just a heads up incase you see me hammering your site. We have controls and weights so we won't just kill your servers :)

 

pmkpmk




msg:1528534
 9:18 pm on Apr 1, 2004 (gmt 0)

And what is it spidering for? What does it intend to build?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved