Mozdex

URL in user agent is 404

[GaryK]

Mozdex/0.7.2-dev+(Mozdex;+http://www.mozdex.com/bot.html;[e-mail])

63.246.154.32

Read and respected robots.txt. But I don't like bots that have phony URLs; it's worse than no URL, IMO!

[jdMorgan]

Ref: [webmasterworld.com...] msg#4

Jim

[wilderness]

I have the following notes about mozdex:

1)63.246.154.32 - - [31/Mar/2005:20:54:43 -0800] "GET /robots.txt HTTP/1.0"
200 3330 "-" "Mozdex/0.06-dev (Mozdex; [mozdex.com...]
[e-mail])"

2) 208.37.26.120 - - [29/Jun/2004:13:40:25 -0700] "GET /robots.txt HTTP/1.0"
403 - "-" "mozDex/0.05-dev (mozDex; [mozdex.com...]
[e-mail])"

In thw April visit, it was grabbing primarily DMOZ pages.
One of these is from a colocator IP and I rarely submit these types to this forum.

Don

[GaryK]

Thanks guys.

Jim, is ByronM claiming to be the person responsible for Mozdex? It seems like it, but my thinking is a bit off these days from all the pain meds following my cancer surgery.

If he is the owner, the e-mail address in his user agent is bad; just like the URL. I tried to e-mail him and got a permanent failure error:

PERM_FAILURE: SMTP Error (state 9): 550-"The recipient cannot be verified."

I'll wait until I get a reply from you before I sticky him.

[anallawalla]

I get a "500 Servlet Exception" if I search with Mozdex. I am one of the donors but I don't think I gave enough :)

[GaryK]

I'm getting the same error. I was going to contact the owner today anyway via a sticky about the incorrect contact information. I'll be sure to mention this error to him. :)

[ByronM]

[e-mail] still bounces, we will try and figure out a way to get emails through that aren't spam. (for every batch of 100k sites we spider we probably get 10x that in spam back to the spider address..)

As far as the site, just drop a trouble ticket under contact us if you have problems :)

and yes, i just use general co-location facilities.. this is an out of my pocket project for now.

[jdMorgan]

It might reduce your spam-bounce load to use "spider at mozdex dot com" in the UA... Like the Fast Webcrawler used to:

"FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)"

Jim