Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: goodroi
Just a quick question regarding the robots.txt file. From everything I have read I gather that it is primarily used as excluding spiders from certain directories and specific pages. Is it possible to use it to allow spiders. I just want it on the site incase spiders look for it but I don't want to exclude the spiders from any areas in the site.
Therefore is there such a thing as the following?
If so will this give access to all areas of the site.
Thanks in advance for anyone able to help :)
BTW- The site is well indexed by most search engines, big and small. I don't even have to resubmit anymore. Makes my job easier.