| mozDex is spidering
FYI on the user agent |
ByronM
#:1528533 | 9:12 pm on Apr 1, 2004 (utc 0) |
Our bot follows the robots.txt exclusion standard, which is described at [robotstxt.org...] Depending on the configuration, our robot may obey different rules. To make it simple to send our bot away, we'll always obey rules for "mozDex". Here are the different cases.
When we're running to populate our index, we'll advertise the agent "mozDex", and obey rules for "mozDex" if they exist, or "mozdex", or "*". To ban all bots from your site, place the following in your robots.txt file: User-agent: * Disallow: / To ban mozDex bots from your site, place the following in your robots.txt file: User-agent: mozDex Disallow: /
Just a heads up incase you see me hammering your site. We have controls and weights so we won't just kill your servers :)
|
pmkpmk
#:1528534 | 9:18 pm on Apr 1, 2004 (utc 0) |
And what is it spidering for? What does it intend to build?
|
|
