Some days ago I found somewhere in some Webmasterworld forum a thread with posts on using mod_rewrite to stop spiders/robots. But I can't find it again! Would you tell me where is it? Thanks!
lazerzubb
6:56 pm on May 29, 2002 (gmt 0)
I suggest that you use a robots.txt file instead, yes there is some spiders which wont pick it up, but all the legitimate spiders will read more at www.robotstxt.org
transistor
7:04 pm on May 29, 2002 (gmt 0)
Thank you lazerzubb, I am currently using the robots.txt file, but I want to be much more strict with bad robots/spiders that skip this file.
