Welcome to WebmasterWorld Guest from 54.224.253.82

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

LoneStarBot Doesn't Honor Robots.txt

Yippy Ki Yay

     
2:41 am on Jun 2, 2012 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14646
votes: 93


Asked for Robots.txt, was denied, repeatedly tried to access site anyway.

69-195-71-70.bluehost.com
User-Agent: Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)

69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"

Guess if you ask more than once you expect different results?

From the website: [setxwebdev.com...]

About the LoneStarBot

LoneStarBot has moved from WebshoppeSolutions Dedicated ip address 67.20.109.179 to ..

Dedicated ip address: IP Information for 69.195.71.70

User Agent: Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm).

LoneStarBot obeys robots.txt and NOINDEX - NOFOLLOW tags.

LoneStarBot parses and indexes websites that are Information Technology related.


Sorry guys, sounds good but I have log files that prove otherwise.
3:51 am on June 2, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5437
votes: 2


Here's the rest of Bluehost.
Thanks for the heads up, as I only had two of these previously.

RewriteCond %{REMOTE_ADDR} ^173\.254\.([0-9]|[1-9][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^50\.87\. [OR]
RewriteCond %{REMOTE_ADDR} ^66\.147\.(2[45][0-9])\. [OR]
RewriteCond %{REMOTE_ADDR} ^67\.20\.(6[4-9]|[789][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^67\.222\.(3[2-9]|[45][0-9]|6[0-3])\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.195\.(6[4-9]|[789][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.89\.(1[5-9]|2[0-9]|3[01])\. [OR]
RewriteCond %{REMOTE_ADDR} ^70\.40\.(19[2-9]|2[01][0-9]|22[0-3])\. [OR]
RewriteCond %{REMOTE_ADDR} ^74\.220\.(19[2-9]|2[01][0-9]|22[0-3])\. [OR]
6:58 pm on June 2, 2012 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3101
votes: 3


Thanks. I didn't have all of those ranges.

I have a note that 69.89.16.0 - 69.89.31.255 is bluehost but 69.89.0.0 - 69.89.15.255 is Strategic Systems Consulting (also blocked) - you include 69.89.15 in the rewrite.
3:56 pm on June 7, 2012 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:July 23, 2004
posts:480
votes: 35


Oh crap @Incredibill .. No excuses .. this thing is supposed to obey.

I put on a different write a while back and upon reading this thread realized that I miswrote a modifier.

Apologies all around to the good people of Webmaster world .. Lonestarbot has been around for quite a while and trust me when I say that it is not some rouge entity that roams about unbridaled.

The UA is Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm) .. and the only ip address you should see this coming from is 69.195.71.70.

The ip ranges will change once I load this into it's own boxes and the dedicated ip's will point to me and not bluehost.

Thankyou again @Incredibill for bringing this to the attention of those here at Webmasterworld .. And again, my most sincere apologies.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members