Does anyone know if it is possible for people not to be able to type in an address like www.mysite.com/robots.txt and NOT view the robots.txt file?
sonjay
3:27 pm on Jul 27, 2005 (gmt 0)
It has to be viewable for the spiders to see it.
If you have content that actually needs to be kept private, as opposed to merely not being indexed, you need to password-protect it or take it out of your httpd document directory.
