Forum Moderators: open
Is it worth purchasing known ip search engine addresses and subsequent updates or are the free listings adaquete?
Has anyone been recently burned by just detecting for the user agent?
Is this still a bad idea to just detect the user agent or should you also detect for the IP address?
Say googlebot uses a new IP that you don't have yet. Would it be smart to use the user-agent to catch those?
Any other issues I should think about?
yes don't get me wrong use it to catch the new ips BUT I would not do that with a script. I would use Logs (Somesort of raw log reader) and do it by hand. I prefer doing it by hand as an agent can be spoofed and they you are giving away all your hard work to someone that spoofed your script.
I like to be able to have a refering URL in my log script that allows me to follow up with anything that does not have one. I have yet to see a referering url from any of the major spiders.
One last thing. Always to a Trace Route on any new IP so you can see if you are being spoofed.