Globetrotter - 4:55 pm on Dec 15, 2011 (gmt 0)
I think Iíve got the script to work the way I want it. I needed to change some of code implemented the whitelist and Iíve build a notification option so I get an email each time a bot is blocked. I did receive some mail but not as much as with the bot traps Iíve implemented.
So whatís the next step? Do I need to change my robots.txt to let all the bots know I only accept bing and google? Or should I block every bot who tries to crawl the website? Which might be a lot?
Iíve also have a bot trap, just to detect dumb bots, Iím not sure if my settings are well enough to catch those with the bot script as well. I have to see and compare the data after a couple of days. How would I block those bots when they checkout my bot trap with the same code?
Hopefully you could share some tips so I do not have to reinvent the wheel.