Robot.txt

<meta name="Robots" content="index,follow">

[Trojanfx]

On all my websites I have this meta tag and I am told it is correct.(<meta name="Robots" content="index,follow">) My logs show that last month on 1 site I had 19 failed request for /robots.txt, another had 47, I do not know what this means. I did some research and it seems I am missing the robot.txt folder on my server. I was not aware I had to upload and create this new folder on server. My question is:
A. do I create this folder and name it robot.txt on the server?
B. What do I put in the folder?

Advise appreciated. Thanks

[Lord Majestic]

> A. do I create this folder and name it robot.txt on the server?

No - robots.txt is only valid for root of your site, and note that each subdomain must have its own, ie: www.example.com is different site than example.com from robots.txt perspective (normally they point to the same directory, which will not be the case for normal subdomains).

> B. What do I put in the folder?

robots.txt format of which can be found by searching for "robots txt" on G. :)

[Trojanfx]

I did searching on google as before and found this page
[robotstxt.org...]

Bottom line says, what I have in tag is fine and nothing else is needed. If I want them to spider entire site, then what I have is fine. But, question B. asked what do I put in folder and you direct me to google for that text but they say no text needed. Not understanding.

Same goes for failed requests. Total failed requests for all sites is 119 for /robot.txt. What does that mean? I've found no reading on that.

Time is very precious. I do not want to waste yours. Am I overly concerned about these failed requests for /robot.txt?

Frederick

[PatrickDeese]

<meta name="Robots" content="index,follow">

There is no reason to put this in your meta tags - its just clutter.

If you want to eliminate the failed robots.txt requests, create a txt file with a program like notepad and put in this:

User-Agent: *
Disallow: