Forum Moderators: goodroi
User-agent: Mediapartners-Google*
Disallow:
I do not currently have a robots.txt file, instead I have the following meta tags:
<meta name="robots" content="all"></meta>
<meta name="revisit" content="2 days"></meta>
What should I do for best results?
Also, whatexactly does the "revisit" meta tag do? I have used it on several project with good results, but Im not sure what it really does.
Thanx for any reply.
User-agent: *
Disallow:
Meaning that every robot is allowed to spider your site. And you will have a lot less 404's in your logs.
[robotstxt.org ]
The robots "all" meta means robots are allowed to index the page and follow the links on it. Well they will do that anyway. There's no point in telling a robot what to do. You tell them what not to do.
The "revisit" meta tag -sorry- is really useless.. You can't tell a bot when to return. They come and go whenever they want.
And by the way, there's no need for the closing </meta> tag.