Welcome to WebmasterWorld Guest from 50.19.156.133

Forum Moderators: goodroi

Message Too Old, No Replies

mozDex is spidering

FYI on the user agent

     

ByronM

9:12 pm on Apr 1, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Our bot follows the robots.txt exclusion standard, which is described at [robotstxt.org...] Depending on the configuration, our robot may obey different rules. To make it simple to send our bot away, we'll always obey rules for "mozDex". Here are the different cases.
When we're running to populate our index, we'll advertise the agent "mozDex", and obey rules for "mozDex" if they exist, or "mozdex", or "*".

To ban all bots from your site, place the following in your robots.txt file:

User-agent: *

Disallow: /

To ban mozDex bots from your site, place the following in your robots.txt file:

User-agent: mozDex

Disallow: /

Just a heads up incase you see me hammering your site. We have controls and weights so we won't just kill your servers :)

pmkpmk

9:18 pm on Apr 1, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



And what is it spidering for? What does it intend to build?
 

Featured Threads

Hot Threads This Week

Hot Threads This Month