homepage Welcome to WebmasterWorld Guest from 54.167.179.48
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
BeetleBot
188.207.200.116
Bewenched

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4518888 posted 10:15 pm on Nov 13, 2012 (gmt 0)

Mozilla/5.0+(compatible;+BeetleBot;+)

188.207.200.116

 

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4518888 posted 10:47 pm on Nov 14, 2012 (gmt 0)

49 hits on that IP (and no other in the range 188.207.0.0/16) since April.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4518888 posted 12:08 am on Nov 15, 2012 (gmt 0)

Bewenched could you supply more info? Did it ask for robots.txt? If so did it obey? What type files did it retrieve? How many?

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4518888 posted 9:19 am on Nov 15, 2012 (gmt 0)

188.207.200.116 - - [12/Nov/2012:09:16:34 +0000] "GET /robots.txt HTTP/1.0" 200 2719 "-" "Mozilla/5.0 (compatible; BeetleBot; )"
188.207.200.116 - - [12/Nov/2012:09:16:35 +0000] "GET /MyFolder/MySub/MyPage.html HTTP/1.0" 403 559 "-" "Mozilla/5.0 (compatible; BeetleBot; )"

keyplr,
note the trailing broken space.

FWIW, this thing appeared a very short while after I added the denial for Inktomi, and requesting the same page that Inktomi requested.
It's likely just a coincidence, however the requests were close enough to through up a flag.
This particular page is plentiful with links leading to it, and diverse widget keywords.

jimbeetle

WebmasterWorld Senior Member jimbeetle us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4518888 posted 4:02 pm on Nov 17, 2012 (gmt 0)

Oops!

Bewenched

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4518888 posted 1:47 am on Nov 18, 2012 (gmt 0)

Interesting that it showed up after inktomi for you... I haven't blocked inktomi at all, but this thing showed up, did request robots.... I've blocked it now... it will be interesting if it complies with robots.txt.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved