Page is a not externally linkable
Garyh - 3:21 pm on Jan 17, 2001 (gmt 0)
House of Blues said that Streambox's spider ignored the robots.txt file. Streambox has ignored what House of Blues characterizes as a type of "No Trespassing" sign on its web server and use a piece of personal property without the owners permission. At the heart of these cases, someone has simply linked to a file without permission.
Don't have a lot of time today to expound, but just saw this interesting article about a spider ignoring the robots.txt file. This months Webtechniques, Feb. 2001 page 18 talks about "House of Blues" filing a suit against Streambox over "Streamlinking" issues. [webtechniques.com...]