Welcome to WebmasterWorld Guest from 54.166.180.246

Forum Moderators: goodroi

Message Too Old, No Replies

Syntax: User-agent: Mediapartners-Google*

     
2:34 pm on Nov 19, 2004 (gmt 0)

New User

10+ Year Member

joined:Nov 9, 2004
posts:13
votes: 0


Hi, I've just taken a new webmaster job and was looking at the previous webmaster's robots.txt file today. The file contains the following code:

User-agent: Mediapartners-Google*
Disallow: /cgi-bin/

(with many more directories disallowed).

In this syntax, does the star at the end of the user-agent line still disallow all other robots from the following directories?

Does adding the "Mediapartners-Google" bit allow that robot anything, or specifically disallow the robot from the following directories?

The old webmaster seems to think this allows google more access, but I've never encountered that syntax before and just want to be sure.

Thank You!
JHUBB

2:45 pm on Nov 19, 2004 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9068
votes: 4


robots.txt can only be used to block robots, not to allow them - so in your example, the mediapartners-google bot (which is the one used for AdSense, not the indexing robot) is banned from accessing the list of directories. However, the rule applies only to that particular bot.

If you want to exclude all bots from that list of directories, then you simply need:

User-agent: *
Disallow: /cgi-bin/
3:30 pm on Nov 19, 2004 (gmt 0)

New User

10+ Year Member

joined:Nov 9, 2004
posts:13
votes: 0


Thanks, that's what I was thinking. Apparently the google line is required for the Real Cities advertising we do, but I don't know how it helps google ad sense with anything since it is just disallowing it.

I'm still looking for clarification, though, on the * that appears in the line "User-agent: Mediapartners-Google*"

Does that star add all the other bots to the disallows? If not, what service does it provide in this syntax?

Thanks,
jhubb

3:40 pm on Nov 19, 2004 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9068
votes: 4


The star is simply a wildcard for the mediapartners-google bots. The exact version at the momment is Mediapartners-Google/2.1 (I think), but the rule blocks that and any later versions too.
3:21 pm on Nov 22, 2004 (gmt 0)

New User

10+ Year Member

joined:Nov 9, 2004
posts:13
votes: 0


So, if I put in my robots.txt:

"
User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /cgi-bin/
"

Would this allow Mediapartners-Google into all directories, while restricting other user-agents? From what I can understand from the previous webmaster, this is the intent.