Redirecting formmail hunters to spider traip - getout.pl

Can I be down? How?

[cybertime]

I have noticed an increase in the amount of formmail script hunters on my server log. I am currently using getout.pl - [webmasterworld.com...]

I would like to redirect requests for formmail.pl,sendmail.pl, mailform.pl and all other variations to the spider trap – getout.pl.

Is this possible? How can I do it?

My .htaccess file currently looks like:

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
# Block bad-bots using lines written by bad_bot.pl script above
SetEnvIf Request_URI "^(/403.*\.html¦/robots\.txt)$" allowit
<Files *>
order deny,allow
deny from env=ban
allow from env=allowit
</Files>

AuthName www.mysite.com
AuthUserFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.pwd
AuthGroupFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.grp

RedirectMatch (.*)\.html$ [mysite.com$1.htm...]

DirectoryIndex index.htm

RewriteCond %{REQUEST_URI} (mail.?form¦form¦form.?mail¦mail¦mailto)\.(cgi¦exe¦pl¦asp¦php)$ [NC,OR]
RewriteRule sendmail\.(pl¦cgi) [mysite.com...] [NC,L]

ErrorDocument 301 [mysite.com...]
ErrorDocument 403 [mysite.com...]
ErrorDocument 404 [mysite.com...]
ErrorDocument 410 [mysite.com...]
ErrorDocument 500 [mysite.com...]

Thank you for your assistance.

[jdMorgan]

There are several rather large errors in the existing trap code. Delete the RewriteCond and try this instead:

Options +FollowSymLinks
RewriteEngine on
RewriteRule (mail.?form¦form¦form.?mail¦mail¦mailto¦[b]sendmail[/b])\.(cgi¦exe¦pl¦asp¦php)$ [b]/cgi[/b]-bin/trap.pl [NC,L]

This will internally (and silently) rewrite requests for mail scripts to trap.pl.

Note that posting on this board changes solid pipe characters to broken "¦" pipes. You must edit the code above to use solid pipes before attempting to use it.

For more information on mod_rewrite, see the resources cited in our charter [webmasterworld.com].

Jim

[Storyman]

Spidertime,

Can you provide a link to getout.pl?

Googled and came up blank.

[cybertime]

Thank you jdMorgan.

StoryMan the links are:

[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]

[cybertime]

jdMorgan,

I think I have done something wrong with <b>getout.php</b>(I am using the getout.php in my html folder and the trap.pl in the cgi folder) script or my .htaccess file.

I believe the spider trap is to prevent the spider from accessing other files and trap/ban it but that is not happening.

Files are being downloaded after the getout.php script runs and I received an e-mails stating that:

The following ip just got banned because it accessed the spider trap.

111.111.1111
Mozilla/4.0 (compatible; MSIE 6.0; Windows XP)
[mysite.com...]

Help please.

Thank you.

PS: This is the content of the .htaccess file:

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
SetEnvIf Request_URI "^(/403.*\.htm¦/robots\.txt)$" allowsome
<Files *>
order deny,allow
deny from env=getout
allow from env=allowsome
</Files>
# Block bad-bots using lines written by bad_bot.pl script above
SetEnvIf Request_URI "^(/403.*\.html¦/robots\.txt)$" allowit
<Files *>
order deny,allow
deny from env=ban
allow from env=allowit
</Files>

AuthName www.mysite.com
AuthUserFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.pwd
AuthGroupFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.grp

RedirectMatch (.*)\.html$ [mysite.com$1.htm...]

DirectoryIndex index.htm

Options +FollowSymLinks
RewriteEngine on
RewriteRule (mail.?form¦form¦form.?mail¦mail¦mailto¦sendmail)\.(cgi¦exe¦pl¦asp¦php)$ /cgi-bin/trap.pl [NC,L]

ErrorDocument 301 [mysite.com...]
ErrorDocument 403 [mysite.com...]
ErrorDocument 404 [mysite.com...]
ErrorDocument 410 [mysite.com...]
ErrorDocument 500 [mysite.com...]

[jdMorgan]

You have conflicting Order and Files directives in your code; Therefore some sections will be ignored. Delete those sections and replace them with:

SetEnvIf Request_URI "^(/403.*\.html?¦/robots\.txt)$" allowit
# Block bad-bots using lines written by bad_bot.pl script above
<Files *>
Order deny,allow
Deny from env=ban
Deny from env=getout
Allow from env=allowit
</Files>

The function of limiting HTTP methods should be moved to the mod_rewrite section, so that you do not allow PUTs and DELETEs of 403.htm and robots.txt; Add the code:

RewriteCond %{HTTP_METHOD} ^(PUT¦DELETE¦CONNECT)$ [OR]
RewriteCond %{REQUEST_URI} ^\.ht
RewriteRule .* - [F]

immediately following your RewriteEngine on directive.

Your ErrorDocument directives are also malformed, and will not return the proper HTTP server status code. They will all return 302 redirects instead. Please see the Apache ErrorDocument documentation. The correct directive syntax does not include "http://mysite.com":

ErrorDocument 404 /sitemap.htm
ErrorDocument 410 /sitemap.htm

I *strongly* suggest that you do not use Errordocument to handle 301, 302, 403, or 500 responses. You will confuse your visitors' browsers and search engine robots. This can badly affect your listings in search engine results.

Be aware that using FrontPage extensions and custom .htaccess files together does not work. You can use FrontPage, but if you enable and use the extensions on your server, you can expect problems with .htaccess.

Change all broken pipe "¦" characters in the code above to solid pipes before use.

Jim

[cybertime]

Thank you very much for your assitance jdMorgan.

I have made the correction as advised.

My htaccess file now looks like:

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

SetEnvIf Request_URI "^(/403.*\.html?:/robots\.txt)$" allowit
# Block bad-bots using lines written by bad_bot.pl script above
<Files *>
Order deny,allow
Deny from env=ban
Deny from env=getout
Allow from env=allowit
</Files>

AuthName www.mysite.com
AuthUserFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.pwd
AuthGroupFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.grp

RedirectMatch (.*)\.html$ [mysite.com$1.htm...]

DirectoryIndex index.htm

Options +FollowSymLinks
RewriteEngine on
RewriteRule (mail.?form¦form¦form.?mail¦mail¦mailto¦sendmail)\.(cgi¦exe¦pl¦asp¦php)$ /cgi-bin/trap.pl [NC,L]

RewriteCond %{HTTP_METHOD} ^(PUT¦DELETE¦CONNECT)$ [OR]
RewriteCond %{REQUEST_URI} ^\.ht
RewriteRule .* - [F]

ErrorDocument 404 /sitemap.htm
ErrorDocument 410 /sitemap.htm

Is the htaccess okay now?

Thanks again.

[jdMorgan]

Looks OK to me. Test it and let us know...

Jim

[cybertime]

Thank you jdMorgan again for your assitance.

I tried to run the script and received the following messages and I was not banned from other pages:

Warning: fopen(): Unable to access /var/www/html/.htaccess in /home/virtual/site26/fst/var/www/html/getout.php on line 4

Warning: fopen(/var/www/html/.htaccess): failed to open stream: No such file or directory in /home/virtual/site26/fst/var/www/html/getout.php on line 4

Warning: filesize(): SAFE MODE Restriction in effect. The script whose uid is 527 is not allowed to access /var/www/html owned by uid 0 in /home/virtual/site26/fst/var/www/html/getout.php on line 5

Warning: fread(): supplied argument is not a valid stream resource in /home/virtual/site26/fst/var/www/html/getout.php on line 5

Warning: fclose(): supplied argument is not a valid stream resource in /home/virtual/site26/fst/var/www/html/getout.php on line 6

Warning: fopen(): SAFE MODE Restriction in effect. The script whose uid is 527 is not allowed to access /var/www/html owned by uid 0 in /home/virtual/site26/fst/var/www/html/getout.php on line 7

Warning: fopen(/var/www/html/.htaccess): failed to open stream: No such file or directory in /home/virtual/site26/fst/var/www/html/getout.php on line 7

Warning: fwrite(): supplied argument is not a valid stream resource in /home/virtual/site26/fst/var/www/html/getout.php on line 8

Warning: fclose(): supplied argument is not a valid stream resource in /home/virtual/site26/fst/var/www/html/getout.php on line 9

This is the script I am using - getout.php:

<?php
$filename = "/var/www/html/.htaccess";
$content = "SetEnvIf Remote_Addr ^".str_replace(".","\.",$_SERVER["REMOTE_ADDR"])."$ getout\r\n";
$handle = fopen($filename, 'r');
$content .= fread($handle,filesize($filename));
fclose($handle);
$handle = fopen($filename, 'w+');
fwrite($handle, $content,strlen($content));
fclose($handle);
mail("[e-mail]",
"Spider Alert!",
"The following ip just got banned because it accessed the spider trap.\r\n\r\n".$_SERVER["REMOTE_ADDR"]."\r\n".$_SERVER["HTTP_USER_AGENT"]."\r\n".$_SERVER["HTTP_REFERER"]
,"FROM: [e-mail]");

// start free emails for spider
$page = '';
for ( $i = 0; $i < 5000; $i++ )
{
$page .= new_email();
}

function new_email()
{
$email = '';
$letters_array = array('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r',
's', 't', 'u', 'v', 'w', 'x', 'y', 'z');
for ( $i = 0; $i < 17; $i++ )
{
$email .= ( $i!== 10 )? $letters_array[ mt_rand( 0, 25) ] : '@';
}
$email .= '.com';
$email = '<a href="mailto:' . $email . '">' . $email . "</a>\n";
return $email;

}

$page .= "Goodbye!";
echo $page;
?>

Thanks again for your assitance.

[edited by: jdMorgan at 8:20 pm (utc) on Oct. 13, 2004]
[edit reason] Removed specifics per TOS [/edit]

[jdMorgan]

It is likely that you have not set the file permissions on your .htaccess file to allow the script to write to it. You will need to set the .htaccess file to Owner-writable, or possibly even World-writable in order to allow the script to write to it.

If you're not familiar with unix file permissions, just do a search on "chmod" -- there are toms of copies of the documentation on-line. You should be able to change the file permissions with your FTP client, or by using TelNet to log into your server at a unix command line.

Jim

[cybertime]

Thank you once again Jim for your help. It is very much appreciated.

I have made the changes as recommended but I am still receiving the following error message:

Warning: rmdir(/home/virtual/site26/fst/var/www/html/trap): Permission denied in /home/virtual/site26/fst/var/www/html/getout.php on line 60

I am using this script - [webmasterworld.com...]

Thanks again for your assitance and I hope this will be my last question.

[jdMorgan]

I don't know much about that script, but the problem is explicitly described in that error message:
"Warning: rmdir(/home/virtual/site26/fst/var/www/html/trap): Permission denied in /home/virtual/site26/fst/var/www/html/getout.php on line 60"

Whatever is on line 60 is failing, and it's failing due to file permissions. You might want to post in the PHP forum about this if you have set the .htaccess file permissions to allow writing and it still fails...

Jim

[cybertime]

Thanks Jim for your help.

I will post the question on the php forum.

Tayo