Welcome to WebmasterWorld Guest from 50.16.112.199

Forum Moderators: goodroi

mozDex is spidering

FYI on the user agent

   
9:12 pm on Apr 1, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Our bot follows the robots.txt exclusion standard, which is described at [robotstxt.org...] Depending on the configuration, our robot may obey different rules. To make it simple to send our bot away, we'll always obey rules for "mozDex". Here are the different cases.
When we're running to populate our index, we'll advertise the agent "mozDex", and obey rules for "mozDex" if they exist, or "mozdex", or "*".

To ban all bots from your site, place the following in your robots.txt file:

User-agent: *

Disallow: /

To ban mozDex bots from your site, place the following in your robots.txt file:

User-agent: mozDex

Disallow: /

Just a heads up incase you see me hammering your site. We have controls and weights so we won't just kill your servers :)

9:18 pm on Apr 1, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



And what is it spidering for? What does it intend to build?
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month