Welcome to WebmasterWorld Guest from 126.96.36.199
Forum Moderators: goodroi
House of Blues said that Streambox's spider ignored the robots.txt file. Streambox has ignored what House of Blues characterizes as a type of "No Trespassing" sign on its web server and use a piece of personal property without the owners permission.
At the heart of these cases, someone has simply linked to a file without permission.
I doubt it. I dont know where I heard it, but what I heard was that the best of them indexed a mere 25-30% of all pages out there.
Ofc, if there's any commercial interest in your site, you'll look yourself to get it listed.
Just background info.