Welcome to WebmasterWorld Guest from 50.19.156.19

Forum Moderators: goodroi

Message Too Old, No Replies

Allowing Adsence bot

Googled emailed me asking to allow the adsence bot

     

alahamdan

7:32 pm on Dec 2, 2008 (gmt 0)

5+ Year Member



Hello

I used to use this robots.txt file:

User-agent: *
Disallow: /forum/adm/
Disallow: /forum/docs/
Disallow: /forum/download/file.php
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/language/
Disallow: /forum/mcp.php
Disallow: /forum/posting.php
Disallow: /forum/ucp.php
Disallow: /forum/search.php
Disallow: /forum/styles/
Disallow: /forum/viewonline.php
Disallow: /forum/style.php
Disallow: /forum/style.php*

Now i got an email from adsence team, that the bot cant crawl some of my pages, they asked me to add the following to robots.txt:

User-agent: Mediapartners-Google*
Disallow:

So, what i want now, is to allow adsence bot to crawl all my site, while preventing all other bots from crawling the listed folders above! how should the robots file looks like?

i used this :

User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: /forum/adm/
Disallow: /forum/docs/
Disallow: /forum/download/file.php
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/language/
Disallow: /forum/mcp.php
Disallow: /forum/posting.php
Disallow: /forum/ucp.php
Disallow: /forum/search.php
Disallow: /forum/styles/
Disallow: /forum/viewonline.php
Disallow: /forum/style.php
Disallow: /forum/style.php*

This is the correct way? or im mistaken?

Thanx in advance.

goodroi

1:18 pm on Dec 5, 2008 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



That will work for Google. Technically speaking I believe you are missing some blank spaces.

alahamdan

1:29 pm on Dec 5, 2008 (gmt 0)

5+ Year Member



Thanx for replay

what do you mean by im missing some blank spaced please?

Receptional Andy

1:59 pm on Dec 5, 2008 (gmt 0)



You should have a blank line for your second user-agent line, and I believe one right at the end.

Also, why the wildcard in the user-agent? I don't believe this is necessary. You can also drop the second of these two lines, since robots.txt uses prefix matching:

Disallow: /forum/style.php
Disallow: /forum/style.php*

alahamdan

2:02 pm on Dec 5, 2008 (gmt 0)

5+ Year Member



Yes i got your idea i will fix it now

Thanks very much for your help