Forum Moderators: open
66.62.122.69 - - [26/Feb/2005:20:59:37 -0800] "GET /robots.txt HTTP/1.1" 200 1955 "-" "Mozilla/5.0 CostaCider Search [costacider.com...]Other than the root url and the bot page, all the others render 404s. That's not good.
Expecting me to either dl or open any kind of file is literally stupid.
I've banned this puppy by both IP Number and robots.txt that it says it obeys. However, it fails to say how often that text will be requested.
<blech!>
66.62.122.69 - - [10/Mar/2005:10:52:56 -0800] "GET / HTTP/1.1" 403 20407 "-" "Mozilla/5.0 CostaCider Search [costacider.com...]Does NOT respect robots.txt!
Their 'system down' now may be a result of this.
I banned them as well.