homepage Welcome to WebmasterWorld Guest from 54.211.190.232
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Twiceler
no website
bose




msg:404874
 4:38 pm on May 29, 2005 (gmt 0)

Spotted the following in my logs a couple of days ago:

64.62.136.201 - - [26/May/2005:13:05:36 -0500] "GET /robots.txt HTTP/1.0" 200 204 "-" "Twiceler www.cuill.com/robots.html"

Using that Hurricane Electric IP, it fetched robots.txt a few times since then. Their website is not reachable.

 

pendanticist




msg:404875
 12:05 am on May 31, 2005 (gmt 0)

It has been my experience this bot pretty much did what it wanted to do, until I widened the scope a bit.

64.62.136

This one utilizes the entire last octet.

bose




msg:404876
 2:16 am on May 31, 2005 (gmt 0)

Thanks for the feedback, pendanticist.

I have now blocked that range.

wilderness




msg:404877
 8:14 pm on May 31, 2005 (gmt 0)

You'd save yourself quite a few pests if you expaned the line to:

RewriteCond %{REMOTE_ADDR} ^64\.62\.(12[8-9]¦1[3-9][0-9]¦2[0-5][0-9])\. [OR]

bose




msg:404878
 5:07 pm on Jun 1, 2005 (gmt 0)

Thanks for your tip, wilderness. Not sure if I have seen any others from that IP range yet, but I'll take your word for it. :)

What a joy to swat a bunch of those unwashed crawlers in one broad stroke!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved