homepage Welcome to WebmasterWorld Guest from 54.166.173.147
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
BeetleBot
188.207.200.116
Bewenched




msg:4518890
 10:15 pm on Nov 13, 2012 (gmt 0)

Mozilla/5.0+(compatible;+BeetleBot;+)

188.207.200.116

 

dstiles




msg:4519181
 10:47 pm on Nov 14, 2012 (gmt 0)

49 hits on that IP (and no other in the range 188.207.0.0/16) since April.

keyplyr




msg:4519200
 12:08 am on Nov 15, 2012 (gmt 0)

Bewenched could you supply more info? Did it ask for robots.txt? If so did it obey? What type files did it retrieve? How many?

wilderness




msg:4519288
 9:19 am on Nov 15, 2012 (gmt 0)

188.207.200.116 - - [12/Nov/2012:09:16:34 +0000] "GET /robots.txt HTTP/1.0" 200 2719 "-" "Mozilla/5.0 (compatible; BeetleBot; )"
188.207.200.116 - - [12/Nov/2012:09:16:35 +0000] "GET /MyFolder/MySub/MyPage.html HTTP/1.0" 403 559 "-" "Mozilla/5.0 (compatible; BeetleBot; )"

keyplr,
note the trailing broken space.

FWIW, this thing appeared a very short while after I added the denial for Inktomi, and requesting the same page that Inktomi requested.
It's likely just a coincidence, however the requests were close enough to through up a flag.
This particular page is plentiful with links leading to it, and diverse widget keywords.

jimbeetle




msg:4519909
 4:02 pm on Nov 17, 2012 (gmt 0)

Oops!

Bewenched




msg:4520004
 1:47 am on Nov 18, 2012 (gmt 0)

Interesting that it showed up after inktomi for you... I haven't blocked inktomi at all, but this thing showed up, did request robots.... I've blocked it now... it will be interesting if it complies with robots.txt.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved