-- Search Engine Spider and User Agent Identification
Lord_Majestic - 3:01 pm on Jun 25, 2009 (gmt 0)
In any case relying on robots.txt to determine if a bot is genuine
If bot obeys your robots.txt then in my it is a good legit bot in its own right.
Incidentally, at the time of writing your site does not respond and according to robtex there is no A record for it.
We had 20 minutes downtime this night from 2:24 to 2:44 am GMT.
Whether it was genuine or not I had no idea and didn't care. It was blocked then and remains blocked now, as are all distributed bots.
We respect your block (I hope you allow us retrieve robots.txt to see it). What we say is that we can't be held responsible for actions of other people who have nothing to do with us and just use faked user-agents: I get email spam every day with fake user-agents but I won't take it out on users whose email addresses were faked.