Forum Moderators: open

Message Too Old, No Replies

Click-thru abuse or log spammer?

         

wilderness

11:21 pm on Jul 24, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Obviously not what Majestic intended his bot to used for!

72.90.71.zzz - - [23/Jul/2008:11:42:04 -0500] "GET /robots.txt HTTP/1.0" 200 4740 "http://majestic12.co.uk/bot.php?I=zzzzzzzzz4D335584B75DF04ECBB015C-64F40E1A0E171A75-2E6DA6BCBzzzzzzz-www.MySite.net" "Mozilla/5.0 (compatible; MJ12bot/v1.2.1; [majestic12.co.uk...]
72.90.71.zzz - - [23/Jul/2008:11:42:04 -0500] "GET /MyFolder/MyPage.html HTTP/1.1" 403 - "http://www.example.com/urlclick.php?id=zzzzzzzz&url=http://www.MySite.net/MyFolder/MyPage.html" "Mozilla/5.0 (compatible; MJ12bot/v1.2.1; [majestic12.co.uk...]

dstiles

2:19 am on Jul 25, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I've banned all MJ12bot. They come in seriously large numbers at a very high rate from unpredictable IPs. Usually any MJ12 IP of more than a few dozen hits per minute gets firewalled (as much as one can in IIS).

Staffa

7:35 am on Jul 25, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have it banned as well, it ignores robots.txt

Megaclinium

3:38 pm on Jul 28, 2008 (gmt 0)

10+ Year Member



I've seen them too hit my site sporadically. Seems to be quite stupid as it tries to retrieve pages that don't exist.

wilderness

4:06 pm on Jul 28, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



For clarification!

My heads up wasn't about the majestic bot in general,
rather. . . .

this particular use of add-on click thru's and the second log line.

In Majestic's defense (while eating my hat and gagging), I seem to recall that his bot utilizes invalid 404's (similar to Slurp and others) to confirm what a valid 404's should return.

dstiles

8:41 pm on Jul 28, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Ahh. Missed that, sorry. Still banned, though. :)