homepage Welcome to WebmasterWorld Guest from 23.22.29.137
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
CrawlDaddy abot Site Harvester
WarmGlow




msg:4613494
 7:00 pm on Sep 28, 2013 (gmt 0)

Did not request robots.txt file.
Banned itself by disobeying robots.txt standard directives.

Sample access log entry:
64.202.161.41 - - [27/Sep/2013:23:59:14 -0400] "GET / HTTP/1.1" 403 362 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; CrawlDaddy v0.3.0 abot v1.2.0.0 http*://code.google.com/p/abot)"
Note: Asterisk (*) added by me to disable hot linking the above URL in this message.

Came from two remote host IPs:
64.202.161.41
64.202.161.46


The remote host IPs are owned by GoDaddy.com, LLC (GODAD).

The user of the two remote host IPs also attempted to find WordPress vulnerabilities. Here is a sample access log entry:
64.202.161.41 - - [27/Sep/2013:23:59:12 -0400] "GET /wp-login.php HTTP/1.1" 403 366 "-" "User-Agent\tMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"

HTH . . .

Dennis

 

not2easy




msg:4613528
 10:56 pm on Sep 28, 2013 (gmt 0)

GoDaddy is a Hosting company, they can never send any visitors as they don't provide Internet Services like, say, Verizon does so it is generally safe to block Hosting companies' full range unless some script services you use are hosted there.

I do have them blocked across all sites. From your observations on what they were doing, I'd just block the whole 64.202.160.0/19 GODaddy range.

keyplyr




msg:4613540
 11:37 pm on Sep 28, 2013 (gmt 0)

Report their behavior to GoDaddy.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved