homepage Welcome to WebmasterWorld Guest from 54.145.252.85
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Twiceler
no website
bose

10+ Year Member



 
Msg#: 2855 posted 4:38 pm on May 29, 2005 (gmt 0)

Spotted the following in my logs a couple of days ago:

64.62.136.201 - - [26/May/2005:13:05:36 -0500] "GET /robots.txt HTTP/1.0" 200 204 "-" "Twiceler www.cuill.com/robots.html"

Using that Hurricane Electric IP, it fetched robots.txt a few times since then. Their website is not reachable.

 

pendanticist

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2855 posted 12:05 am on May 31, 2005 (gmt 0)

It has been my experience this bot pretty much did what it wanted to do, until I widened the scope a bit.

64.62.136

This one utilizes the entire last octet.

bose

10+ Year Member



 
Msg#: 2855 posted 2:16 am on May 31, 2005 (gmt 0)

Thanks for the feedback, pendanticist.

I have now blocked that range.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 2855 posted 8:14 pm on May 31, 2005 (gmt 0)

You'd save yourself quite a few pests if you expaned the line to:

RewriteCond %{REMOTE_ADDR} ^64\.62\.(12[8-9]¦1[3-9][0-9]¦2[0-5][0-9])\. [OR]

bose

10+ Year Member



 
Msg#: 2855 posted 5:07 pm on Jun 1, 2005 (gmt 0)

Thanks for your tip, wilderness. Not sure if I have seen any others from that IP range yet, but I'll take your word for it. :)

What a joy to swat a bunch of those unwashed crawlers in one broad stroke!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved