Forum Moderators: goodroi
I used to use this robots.txt file:
User-agent: *
Disallow: /forum/adm/
Disallow: /forum/docs/
Disallow: /forum/download/file.php
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/language/
Disallow: /forum/mcp.php
Disallow: /forum/posting.php
Disallow: /forum/ucp.php
Disallow: /forum/search.php
Disallow: /forum/styles/
Disallow: /forum/viewonline.php
Disallow: /forum/style.php
Disallow: /forum/style.php*
Now i got an email from adsence team, that the bot cant crawl some of my pages, they asked me to add the following to robots.txt:
User-agent: Mediapartners-Google*
Disallow:
So, what i want now, is to allow adsence bot to crawl all my site, while preventing all other bots from crawling the listed folders above! how should the robots file looks like?
i used this :
User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: /forum/adm/
Disallow: /forum/docs/
Disallow: /forum/download/file.php
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/language/
Disallow: /forum/mcp.php
Disallow: /forum/posting.php
Disallow: /forum/ucp.php
Disallow: /forum/search.php
Disallow: /forum/styles/
Disallow: /forum/viewonline.php
Disallow: /forum/style.php
Disallow: /forum/style.php*
This is the correct way? or im mistaken?
Thanx in advance.
Also, why the wildcard in the user-agent? I don't believe this is necessary. You can also drop the second of these two lines, since robots.txt uses prefix matching:
Disallow: /forum/style.php
Disallow: /forum/style.php*