RobotMidareru

RobotMidareru/0.7libwww-perl/5.65

[vodkabird]

Rude little swine didn't ask for robots.txt - seems to be coming from 68.168.216.6, but tracert is full of fake rDNS'.

Nothing on Google or WW about this one - anyone else any clearer?

Midareru is Japanese for 'Yearning' apparently. So a bit of culture here too. :)

[wilderness]

Adelphia in PA.

In most instances you'll be better of denying anything which contains "libwww-perl" in th UA.

[jdMorgan]

Unless it's someone you like checking links...

I'm using:

# Block libwww-perl except from AltaVista, Inktomi, and IA Archiver 
RewriteCond %{HTTP_USER_AGENT} libwww-perl/[0-9] [NC]
RewriteCond %{REMOTE_ADDR} !^209\.73\.(1[6-8][0-9]¦19[01])\.
RewriteCond %{REMOTE_ADDR} !^209\.131\.(3[2-9]¦[45][0-9]¦6[0-3])\.
RewriteCond %{REMOTE_ADDR} !^209\.237\.23[2-5]\.
RewriteRule!^(403.*\.html¦robots\.txt)$ - [F]

The rule allows anyone to read robots.txt and my custom 403 pages. Some may not want to allow IA Archiver, but I do.

Jim
<edited>Corrected spaces per following posts.</edit>

[edited by: jdMorgan at 3:09 am (utc) on April 20, 2003]

[Andrue]

Which one of those is ia_archiver? i shut them off a few weeks ago, but i didnt know that altavista used the libwww perl also.

Is inktomi a good or bad bot? i have never been able to figure that one out, looks good but i used to get a lot of spam from my old site and i know inktomi was all over it.

[jdMorgan]

IA is 209.237.232.0 - 209.237.235.255

Jim

[Andrue]

Thanx, btw, i think there is supposed to be a space between the } and the! in the rewrites above.

got some server errors from that.

[wilderness]

<snip>supposed to be a space between the</snip>

Andrue
I'll save Jim some time and tell you the forum server omits spaces in some instances. As well as changing a the "¦" which I entered as a straight line.

For him to make these extra typing precautions is the many lines of code he graciously provides is a big PITA.

[jrobbio]

Just testing the style codes.

# Block libwww-perl except from AltaVista, Inktomi, and IA Archiver 
RewriteCond %{HTTP_USER_AGENT} libwww-perl/[0-9] [NC] 
RewriteCond %{REMOTE_ADDR}!^209\.73\.(1[6-8][0-9]¦19[01])\. 
RewriteCond %{REMOTE_ADDR}!^209\.131\.(3[2-9]¦[45][0-9]¦6[0-3])\. 
RewriteCond %{REMOTE_ADDR}!^209\.237\.23[2-5]\. 
RewriteRule!^(403.*\.html¦robots\.txt)$ - [F] 

Its amazing there is only one forum that deals with all code flawlessly and its hardly known about.