homepage Welcome to WebmasterWorld Guest from 54.163.139.36
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
CrawlDaddy abot Site Harvester
WarmGlow

10+ Year Member



 
Msg#: 4613492 posted 7:00 pm on Sep 28, 2013 (gmt 0)

Did not request robots.txt file.
Banned itself by disobeying robots.txt standard directives.

Sample access log entry:
64.202.161.41 - - [27/Sep/2013:23:59:14 -0400] "GET / HTTP/1.1" 403 362 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; CrawlDaddy v0.3.0 abot v1.2.0.0 http*://code.google.com/p/abot)"
Note: Asterisk (*) added by me to disable hot linking the above URL in this message.

Came from two remote host IPs:
64.202.161.41
64.202.161.46


The remote host IPs are owned by GoDaddy.com, LLC (GODAD).

The user of the two remote host IPs also attempted to find WordPress vulnerabilities. Here is a sample access log entry:
64.202.161.41 - - [27/Sep/2013:23:59:12 -0400] "GET /wp-login.php HTTP/1.1" 403 366 "-" "User-Agent\tMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"

HTH . . .

Dennis

 

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4613492 posted 10:56 pm on Sep 28, 2013 (gmt 0)

GoDaddy is a Hosting company, they can never send any visitors as they don't provide Internet Services like, say, Verizon does so it is generally safe to block Hosting companies' full range unless some script services you use are hosted there.

I do have them blocked across all sites. From your observations on what they were doing, I'd just block the whole 64.202.160.0/19 GODaddy range.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4613492 posted 11:37 pm on Sep 28, 2013 (gmt 0)

Report their behavior to GoDaddy.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved