Forum Moderators: open

Message Too Old, No Replies

BeetleBot

188.207.200.116

         

Bewenched

10:15 pm on Nov 13, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Mozilla/5.0+(compatible;+BeetleBot;+)

188.207.200.116

dstiles

10:47 pm on Nov 14, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



49 hits on that IP (and no other in the range 188.207.0.0/16) since April.

keyplyr

12:08 am on Nov 15, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Bewenched could you supply more info? Did it ask for robots.txt? If so did it obey? What type files did it retrieve? How many?

wilderness

9:19 am on Nov 15, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



188.207.200.116 - - [12/Nov/2012:09:16:34 +0000] "GET /robots.txt HTTP/1.0" 200 2719 "-" "Mozilla/5.0 (compatible; BeetleBot; )"
188.207.200.116 - - [12/Nov/2012:09:16:35 +0000] "GET /MyFolder/MySub/MyPage.html HTTP/1.0" 403 559 "-" "Mozilla/5.0 (compatible; BeetleBot; )"

keyplr,
note the trailing broken space.

FWIW, this thing appeared a very short while after I added the denial for Inktomi, and requesting the same page that Inktomi requested.
It's likely just a coincidence, however the requests were close enough to through up a flag.
This particular page is plentiful with links leading to it, and diverse widget keywords.

jimbeetle

4:02 pm on Nov 17, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Oops!

Bewenched

1:47 am on Nov 18, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Interesting that it showed up after inktomi for you... I haven't blocked inktomi at all, but this thing showed up, did request robots.... I've blocked it now... it will be interesting if it complies with robots.txt.