Forum Moderators: goodroi

Message Too Old, No Replies

Google tells me to change my robots.txt

Should I follow the advice from Google adSense?

         

Bonusbana

1:13 pm on Aug 27, 2004 (gmt 0)

10+ Year Member



When adding an adsense feature on one of my sites, google recommends to optimize the ad revenue by adding this to my robots.txt:

User-agent: Mediapartners-Google*
Disallow:

I do not currently have a robots.txt file, instead I have the following meta tags:

<meta name="robots" content="all"></meta>
<meta name="revisit" content="2 days"></meta>

What should I do for best results?

Also, whatexactly does the "revisit" meta tag do? I have used it on several project with good results, but Im not sure what it really does.

Thanx for any reply.

Span

2:44 pm on Aug 27, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Google probably means not to block the Mediapartners-Google UA from spidering your site.
If you don't have a robots.txt you could make one with these two lines in it:

User-agent: *
Disallow:

Meaning that every robot is allowed to spider your site. And you will have a lot less 404's in your logs.
[robotstxt.org ]

The robots "all" meta means robots are allowed to index the page and follow the links on it. Well they will do that anyway. There's no point in telling a robot what to do. You tell them what not to do.

The "revisit" meta tag -sorry- is really useless.. You can't tell a bot when to return. They come and go whenever they want.

And by the way, there's no need for the closing </meta> tag.