I realize this question boarders on the Spider ID forum that was closed. Although my question is a general one. Is there a way in the robots.txt file to ID spiders that ignore or even skip it? Is one thing to be able to id them when they go to robots.txt first. But what about the ones that skip it?
Any suggestions or direction on where to look for info would be helpful.
one basic idea is to set up a new directory /bottrap/, set a hidden link (probably using a 1x1 transparent gif or some other link invisible for the casual user) on your main page, write the following into your robots.txt User-agent: * Disallow: /bottrap/ and wait watching who is accessing the /bottrap/, either by looking thru your log manually, or by setting up a script /bottrap/index.php sending you an automatic alert.