| LoneStarBot Doesn't Honor Robots.txt
Yippy Ki Yay |
incrediBILL
msg:4460660 | 2:41 am on Jun 2, 2012 (gmt 0) |
Asked for Robots.txt, was denied, repeatedly tried to access site anyway.
69-195-71-70.bluehost.com
User-Agent: Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
69.195.71.70,"Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm)","/index.html"
Guess if you ask more than once you expect different results?
From the website: [setxwebdev.com...]
About the LoneStarBot
LoneStarBot has moved from WebshoppeSolutions Dedicated ip address 67.20.109.179 to ..
Dedicated ip address: IP Information for 69.195.71.70
User Agent: Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm).
LoneStarBot obeys robots.txt and NOINDEX - NOFOLLOW tags.
LoneStarBot parses and indexes websites that are Information Technology related.
|
|
Sorry guys, sounds good but I have log files that prove otherwise.
|
wilderness
msg:4460673 | 3:51 am on Jun 2, 2012 (gmt 0) |
Here's the rest of Bluehost.
Thanks for the heads up, as I only had two of these previously.
RewriteCond %{REMOTE_ADDR} ^173\.254\.([0-9]|[1-9][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^50\.87\. [OR]
RewriteCond %{REMOTE_ADDR} ^66\.147\.(2[45][0-9])\. [OR]
RewriteCond %{REMOTE_ADDR} ^67\.20\.(6[4-9]|[789][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^67\.222\.(3[2-9]|[45][0-9]|6[0-3])\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.195\.(6[4-9]|[789][0-9]|1[01][0-9]|12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.89\.(1[5-9]|2[0-9]|3[01])\. [OR]
RewriteCond %{REMOTE_ADDR} ^70\.40\.(19[2-9]|2[01][0-9]|22[0-3])\. [OR]
RewriteCond %{REMOTE_ADDR} ^74\.220\.(19[2-9]|2[01][0-9]|22[0-3])\. [OR]
|
dstiles
msg:4460799 | 6:58 pm on Jun 2, 2012 (gmt 0) |
Thanks. I didn't have all of those ranges.
I have a note that 69.89.16.0 - 69.89.31.255 is bluehost but 69.89.0.0 - 69.89.15.255 is Strategic Systems Consulting (also blocked) - you include 69.89.15 in the rewrite.
|
mcneely
msg:4462522 | 3:56 pm on Jun 7, 2012 (gmt 0) |
Oh crap @Incredibill .. No excuses .. this thing is supposed to obey.
I put on a different write a while back and upon reading this thread realized that I miswrote a modifier.
Apologies all around to the good people of Webmaster world .. Lonestarbot has been around for quite a while and trust me when I say that it is not some rouge entity that roams about unbridaled.
The UA is Mozilla/5.0 (compatible; LoneStarBot/1.06; +http://www.setxwebdev.com/bot.htm) .. and the only ip address you should see this coming from is 69.195.71.70.
The ip ranges will change once I load this into it's own boxes and the dedicated ip's will point to me and not bluehost.
Thankyou again @Incredibill for bringing this to the attention of those here at Webmasterworld .. And again, my most sincere apologies.
|
|
|
