homepage Welcome to WebmasterWorld Guest from 54.234.59.94
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
LoneStarBot Doesn't Honor Robots.txt
Yippy Ki Yay
incrediBILL




msg:4460660
 2:41 am on Jun 2, 2012 (gmt 0)

Asked for Robots.txt, was denied, repeatedly tried to access site anyway.

69-195-71-70.bluehost.com
User-Agent: Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)

69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"

Guess if you ask more than once you expect different results?

From the website: [setxwebdev.com...]

About the LoneStarBot

LoneStarBot has moved from WebshoppeSolutions Dedicated ip address 67.20.109.179 to ..

Dedicated ip address: IP Information for 69.195.71.70

User Agent: Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm).

LoneStarBot obeys robots.txt and NOINDEX - NOFOLLOW tags.

LoneStarBot parses and indexes websites that are Information Technology related.


Sorry guys, sounds good but I have log files that prove otherwise.

 

wilderness




msg:4460673
 3:51 am on Jun 2, 2012 (gmt 0)

Here's the rest of Bluehost.
Thanks for the heads up, as I only had two of these previously.

RewriteCond %{REMOTE_ADDR} ^173\.254\.([0-9]|[1-9][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^50\.87\. [OR]
RewriteCond %{REMOTE_ADDR} ^66\.147\.(2[45][0-9])\. [OR]
RewriteCond %{REMOTE_ADDR} ^67\.20\.(6[4-9]|[789][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^67\.222\.(3[2-9]|[45][0-9]|6[0-3])\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.195\.(6[4-9]|[789][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.89\.(1[5-9]|2[0-9]|3[01])\. [OR]
RewriteCond %{REMOTE_ADDR} ^70\.40\.(19[2-9]|2[01][0-9]|22[0-3])\. [OR]
RewriteCond %{REMOTE_ADDR} ^74\.220\.(19[2-9]|2[01][0-9]|22[0-3])\. [OR]

dstiles




msg:4460799
 6:58 pm on Jun 2, 2012 (gmt 0)

Thanks. I didn't have all of those ranges.

I have a note that 69.89.16.0 - 69.89.31.255 is bluehost but 69.89.0.0 - 69.89.15.255 is Strategic Systems Consulting (also blocked) - you include 69.89.15 in the rewrite.

mcneely




msg:4462522
 3:56 pm on Jun 7, 2012 (gmt 0)

Oh crap @Incredibill .. No excuses .. this thing is supposed to obey.

I put on a different write a while back and upon reading this thread realized that I miswrote a modifier.

Apologies all around to the good people of Webmaster world .. Lonestarbot has been around for quite a while and trust me when I say that it is not some rouge entity that roams about unbridaled.

The UA is Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm) .. and the only ip address you should see this coming from is 69.195.71.70.

The ip ranges will change once I load this into it's own boxes and the dedicated ip's will point to me and not bluehost.

Thankyou again @Incredibill for bringing this to the attention of those here at Webmasterworld .. And again, my most sincere apologies.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved