Welcome to WebmasterWorld Guest from 54.163.168.15

Forum Moderators: Ocean10000 & incrediBILL

LoneStarBot Doesn't Honor Robots.txt

Yippy Ki Yay

   
2:41 am on Jun 2, 2012 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Asked for Robots.txt, was denied, repeatedly tried to access site anyway.

69-195-71-70.bluehost.com
User-Agent: Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)

69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"

Guess if you ask more than once you expect different results?

From the website: [setxwebdev.com...]

About the LoneStarBot

LoneStarBot has moved from WebshoppeSolutions Dedicated ip address 67.20.109.179 to ..

Dedicated ip address: IP Information for 69.195.71.70

User Agent: Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm).

LoneStarBot obeys robots.txt and NOINDEX - NOFOLLOW tags.

LoneStarBot parses and indexes websites that are Information Technology related.


Sorry guys, sounds good but I have log files that prove otherwise.
3:51 am on Jun 2, 2012 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Here's the rest of Bluehost.
Thanks for the heads up, as I only had two of these previously.

RewriteCond %{REMOTE_ADDR} ^173\.254\.([0-9]|[1-9][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^50\.87\. [OR]
RewriteCond %{REMOTE_ADDR} ^66\.147\.(2[45][0-9])\. [OR]
RewriteCond %{REMOTE_ADDR} ^67\.20\.(6[4-9]|[789][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^67\.222\.(3[2-9]|[45][0-9]|6[0-3])\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.195\.(6[4-9]|[789][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.89\.(1[5-9]|2[0-9]|3[01])\. [OR]
RewriteCond %{REMOTE_ADDR} ^70\.40\.(19[2-9]|2[01][0-9]|22[0-3])\. [OR]
RewriteCond %{REMOTE_ADDR} ^74\.220\.(19[2-9]|2[01][0-9]|22[0-3])\. [OR]
6:58 pm on Jun 2, 2012 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Thanks. I didn't have all of those ranges.

I have a note that 69.89.16.0 - 69.89.31.255 is bluehost but 69.89.0.0 - 69.89.15.255 is Strategic Systems Consulting (also blocked) - you include 69.89.15 in the rewrite.
3:56 pm on Jun 7, 2012 (gmt 0)

10+ Year Member Top Contributors Of The Month



Oh crap @Incredibill .. No excuses .. this thing is supposed to obey.

I put on a different write a while back and upon reading this thread realized that I miswrote a modifier.

Apologies all around to the good people of Webmaster world .. Lonestarbot has been around for quite a while and trust me when I say that it is not some rouge entity that roams about unbridaled.

The UA is Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm) .. and the only ip address you should see this coming from is 69.195.71.70.

The ip ranges will change once I load this into it's own boxes and the dedicated ip's will point to me and not bluehost.

Thankyou again @Incredibill for bringing this to the attention of those here at Webmasterworld .. And again, my most sincere apologies.
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month