212.142.33.101 -- sprung bot trap

[BlueSky]

This guy came by and pulled one page. Two hours later he came back and immediately requested the hidden link to my first bot trap bait. It gave a 302 but didn't spring the trap so I need to figure out what went wrong with that. He then proceeded to take 35 more pages in a totally random order. If it weren't for the fact he was pulling at a regular interval and not downloading graphics/stylesheet he probably could have passed for human. Then for some weird reason, he requested the bot trap itself and got banned. I guess this could have been a delay from the 302, but it was a minute later.

212.142.33.101 - - [26/Oct/2003:16:54:28 -0600] "GET /bot/trap.pl HTTP/1.0" 200 218 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"

He continued requesting pages at a much faster rate of 8-10 per second for another five minutes. I'm going to add a long sleep to my 403 page to slow these guys way down.

Yes, another fine visitor from RIPE:

inetnum: 212.142.33.0 - 212.142.33.255
netname: UPC-BRAIN-2
descr: Brain IP access Amsterdam
country: NL

[jdMorgan]

BlueSky,

A common mistake in badbot baiting is to use an external redirect to forward the bot from the bait faile to the actual trap script. If possible, redirect from the bait file to the actual trap using an internal redirect. In mod_rewrite format, an internal redirect would be:

RewriteRule ^tasty_bait\.html$ /bad/trap.pl [L]

rather than an external redirect:

RewriteRule ^tasty_bait\.html$ http://www.example.com/bad/trap.pl [[b]R[/b],L]

The general problem with using an external redirect is that it requires the user-agent (bad bot) to cooperate, and to re-request the 'page' from the new URL returned in the 301 or 302 redirect response.

Jim

[BlueSky]

You are good. That's exactly what I did. I'm changing it right now. Thanks Jim.