Page is a not externally linkable
incrediBILL - 10:07 pm on Mar 18, 2007 (gmt 0)
Bingo! I agree that in the early days when there were no SE's they had to start somewhere, but that ship sailed about 10 years ago and people SHOULD know they need a robots.txt file by now. It should state right on the "SUBMIT URL" page for the search engines that the only way they can include your site is if you either include a robots.txt or include the appropriate meta tages to prove you want it spidered and someone isn't submitting on your behalf. How the SE's get away without any type of confirmation of ownership blows my mind.
A better analogy is just because you don't have a sign on your front door (robots.txt), doesn't mean that coming inside and stealing your property and giving it away is ok.