LoneStarBot Doesn't Honor Robots.txt

Yippy Ki Yay

[incrediBILL]

Asked for Robots.txt, was denied, repeatedly tried to access site anyway.

69-195-71-70.bluehost.com
User-Agent: Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)

69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"

Guess if you ask more than once you expect different results?

From the website: [setxwebdev.com...]

About the LoneStarBot

LoneStarBot has moved from WebshoppeSolutions Dedicated ip address 67.20.109.179 to ..

Dedicated ip address: IP Information for 69.195.71.70

User Agent: Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm).

LoneStarBot obeys robots.txt and NOINDEX - NOFOLLOW tags.

LoneStarBot parses and indexes websites that are Information Technology related.

Sorry guys, sounds good but I have log files that prove otherwise.

[wilderness]

Here's the rest of Bluehost.
Thanks for the heads up, as I only had two of these previously.

RewriteCond %{REMOTE_ADDR} ^173\.254\.([0-9]|[1-9][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^50\.87\. [OR]
RewriteCond %{REMOTE_ADDR} ^66\.147\.(2[45][0-9])\. [OR]
RewriteCond %{REMOTE_ADDR} ^67\.20\.(6[4-9]|[789][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^67\.222\.(3[2-9]|[45][0-9]|6[0-3])\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.195\.(6[4-9]|[789][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.89\.(1[5-9]|2[0-9]|3[01])\. [OR]
RewriteCond %{REMOTE_ADDR} ^70\.40\.(19[2-9]|2[01][0-9]|22[0-3])\. [OR]
RewriteCond %{REMOTE_ADDR} ^74\.220\.(19[2-9]|2[01][0-9]|22[0-3])\. [OR]

[dstiles]

Thanks. I didn't have all of those ranges.

I have a note that 69.89.16.0 - 69.89.31.255 is bluehost but 69.89.0.0 - 69.89.15.255 is Strategic Systems Consulting (also blocked) - you include 69.89.15 in the rewrite.

[mcneely]

Oh crap @Incredibill .. No excuses .. this thing is supposed to obey.

I put on a different write a while back and upon reading this thread realized that I miswrote a modifier.

Apologies all around to the good people of Webmaster world .. Lonestarbot has been around for quite a while and trust me when I say that it is not some rouge entity that roams about unbridaled.

The UA is Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm) .. and the only ip address you should see this coming from is 69.195.71.70.

The ip ranges will change once I load this into it's own boxes and the dedicated ip's will point to me and not bluehost.

Thankyou again @Incredibill for bringing this to the attention of those here at Webmasterworld .. And again, my most sincere apologies.