Just a quick question regarding the robots.txt file. From everything I have read I gather that it is primarily used as excluding spiders from certain directories and specific pages. Is it possible to use it to allow spiders. I just want it on the site incase spiders look for it but I don't want to exclude the spiders from any areas in the site. Therefore is there such a thing as the following? User-agent: * Allow: /
If so will this give access to all areas of the site.
Yes, but not having one present is much safer. Some brain power challenged robots only look for the presence of a robots.txt and turn around if they find one. There is no "allow" in the official standard, but we've seen so many people use it, I think engines are allowing it.
being the good little seo that i am, i receintly began actually putting something in the robots.txt that we had. just for clarification, are you saying that this could be bad? that, unless necessary, you should not have a robots.txt?