Forum Moderators: open
72.90.71.zzz - - [23/Jul/2008:11:42:04 -0500] "GET /robots.txt HTTP/1.0" 200 4740 "http://majestic12.co.uk/bot.php?I=zzzzzzzzz4D335584B75DF04ECBB015C-64F40E1A0E171A75-2E6DA6BCBzzzzzzz-www.MySite.net" "Mozilla/5.0 (compatible; MJ12bot/v1.2.1; [majestic12.co.uk...]
72.90.71.zzz - - [23/Jul/2008:11:42:04 -0500] "GET /MyFolder/MyPage.html HTTP/1.1" 403 - "http://www.example.com/urlclick.php?id=zzzzzzzz&url=http://www.MySite.net/MyFolder/MyPage.html" "Mozilla/5.0 (compatible; MJ12bot/v1.2.1; [majestic12.co.uk...]
My heads up wasn't about the majestic bot in general,
rather. . . .
this particular use of add-on click thru's and the second log line.
In Majestic's defense (while eating my hat and gagging), I seem to recall that his bot utilizes invalid 404's (similar to Slurp and others) to confirm what a valid 404's should return.