Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
Advise appreciated. Thanks
No - robots.txt is only valid for root of your site, and note that each subdomain must have its own, ie: www.example.com is different site than example.com from robots.txt perspective (normally they point to the same directory, which will not be the case for normal subdomains).
> B. What do I put in the folder?
robots.txt format of which can be found by searching for "robots txt" on G. :)
Bottom line says, what I have in tag is fine and nothing else is needed. If I want them to spider entire site, then what I have is fine. But, question B. asked what do I put in folder and you direct me to google for that text but they say no text needed. Not understanding.
Same goes for failed requests. Total failed requests for all sites is 119 for /robot.txt. What does that mean? I've found no reading on that.
Time is very precious. I do not want to waste yours. Am I overly concerned about these failed requests for /robot.txt?
<meta name="Robots" content="index,follow">
There is no reason to put this in your meta tags - its just clutter.
If you want to eliminate the failed robots.txt requests, create a txt file with a program like notepad and put in this: