Cityreview Robot

strange one. no robots.

[bull]

82.82.146.12 - - [09/Aug/2003:00:18:06 +0200] "GET /odplistedone/ HTTP/1.0" 200 2302 snip "-" "Cityreview Robot <[e-mail]>" "-"

one day later

82.82.148.16 - - [10/Aug/2003:11:36:56 +0200] "GET /odplistedone/indexd.html HTTP/1.0" 200 1376 snip "-" "Cityreview Robot <[e-mail]>" "-" 

82.82.148.16 - - [10/Aug/2003:11:37:05 +0200] "GET /odplistedone/indexe.html HTTP/1.0" 200 1377 snip "-" "Cityreview Robot <[e-mail]>" "-" 

On August 09 wrote an email with a false account to the indicated address. The german respone was that the robot checks for dead links of sites listed in cityview.de ("Cityreview Robot sieht bei den in Cityreview.de eingetragenen Seiten nach, ob diese noch existieren.") and no response to my question on missing robots.txt.

cityreview.de seems to be a DMOZ Clone of Top->World->Deutsch->Regional->Europa->Deutschland, enriched with screenshots of the listed pages.
The requested directory is listed 3 times in the ODP in 3 different cats, but NOT there - the content is totally different and also NOT listed in cityreview.de. Additionally, 82.82.148.* is a large german ADSL provider. And link checking is something different than getting 2 subpages.

Did not like it and 403ed it.

[Yidaki]

The geographical / population data there is at least interesting and afaik unique. Cityreviw is a city ressource compilation backed up with dmoz data. Don't see any problem with this. I'd say it's pretty unfair to discredit them and call them being bad. I don't think they are so dumb to do a disallowed crawl using a real identfier.

Two requests and a standard reply to a faked email account isn't enough to judge, imho. Maybe you should better try contactimg them/him with your real mail account and give them/him some log data to investigate the problem, no!?

[wilderness]

Two requests and a standard reply to a faked email account isn't enough to judge, imho. Maybe you should better try contactimg them/him with your real mail account and give them/him some log data to investigate the problem, no!?

Yidaki
The few sparse lines may not be enough for you?
It was for Bull!
Perhaps you don't monitor your logs to the same extent he does?

I personally find it an absurd waste of my time contacing IP's. The majority only respond by automation.
Since it is the webmaster taking a corrective action?
Perhaps the IP's will some days figure out how to make inquiries as to how and why their ranges were denied?

Until a process exists in a manner receptive to resoultion for these two mediums (webmasters and IP's) than lack of communication, denials and undesired access are inevitable.

[Yidaki]

wilderness,

believe me, i'm one of the most paranoid webmasters out there and i block everything that continously makes disallowed requests and that is using a fakes ip/agent.

But the mentioned bot did identify itself by name/domain/email and miscrediting someone in public based on a few discussable facts can be badly wrong and might hurt someone whose intention was probably not to cheat.

In a ideal world.

[bull]

dumb to do a disallowed crawl

The crawl could not even have been disallowed as robots.txt was not fetched...

miscrediting someone in public

I am telling you simple facts, log excerpts etc., and my personal decision based upon a conflict of what I was told in the email response and the reality in my logs.
If you interpret this as miscrediting, it is your problem.

[Yidaki]

>The crawl could not even have been disallowed as robots.txt was not fetched...

Ok, sorry, i've said it the wrong way: a bot that ignored the robots.txt.

>If you interpret this as miscrediting, it is your problem.

Not a real problem for me. :) But, you're right, it might be a good idea that i remove my pink glasses.