:( there are still have people use win98, definitely
My method is made a spider trap, I cannot give sample by the rule here, basicaly it work like this, I use long time now, work well.
1. make a hidden link on every page, for example
2. you catch all 404 error, and on this 123456 page, you display more links,like
and also put noindex nofollow meta on the page, if a decent robot, will stop here
(I also put Disallow on robots.txt)
3. also catch 404 error, if any one claw the page 1.html 3.html etc,
add the IP to a block list and send 403 Too many users error.
4. all you page must check the ip and if the ip in block list, give 403 Too many users error.
I implement this by IIS and asp, very easy and effective, someday can catch lots of spiders. For example, now my block ip list have
I clean the list every day.
Why send "403 Too many users" error, because if send 404 or 500 error, the spider will know something wrong, and also if your miscatch a good robot, it also has chance to go back.
[edited by: volatilegx at 6:32 pm (utc) on Sep. 19, 2007]
[edit reason] obfuscated ip addresses [/edit]