Page is a not externally linkable
merlin78 - 12:13 am on Apr 3, 2001 (gmt 0)
Just a quick question regarding the robots.txt file. From everything I have read I gather that it is primarily used as excluding spiders from certain directories and specific pages. Is it possible to use it to allow spiders. I just want it on the site incase spiders look for it but I don't want to exclude the spiders from any areas in the site. If so will this give access to all areas of the site. Thanks in advance for anyone able to help :)
Hi,
Therefore is there such a thing as the following?
User-agent: *
Allow: /