homepage Welcome to WebmasterWorld Guest from 54.227.40.166
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Allowing Adsence bot
Googled emailed me asking to allow the adsence bot
alahamdan




msg:3798443
 7:32 pm on Dec 2, 2008 (gmt 0)

Hello

I used to use this robots.txt file:

User-agent: *
Disallow: /forum/adm/
Disallow: /forum/docs/
Disallow: /forum/download/file.php
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/language/
Disallow: /forum/mcp.php
Disallow: /forum/posting.php
Disallow: /forum/ucp.php
Disallow: /forum/search.php
Disallow: /forum/styles/
Disallow: /forum/viewonline.php
Disallow: /forum/style.php
Disallow: /forum/style.php*

Now i got an email from adsence team, that the bot cant crawl some of my pages, they asked me to add the following to robots.txt:

User-agent: Mediapartners-Google*
Disallow:

So, what i want now, is to allow adsence bot to crawl all my site, while preventing all other bots from crawling the listed folders above! how should the robots file looks like?

i used this :

User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: /forum/adm/
Disallow: /forum/docs/
Disallow: /forum/download/file.php
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/language/
Disallow: /forum/mcp.php
Disallow: /forum/posting.php
Disallow: /forum/ucp.php
Disallow: /forum/search.php
Disallow: /forum/styles/
Disallow: /forum/viewonline.php
Disallow: /forum/style.php
Disallow: /forum/style.php*

This is the correct way? or im mistaken?

Thanx in advance.

 

goodroi




msg:3800791
 1:18 pm on Dec 5, 2008 (gmt 0)

That will work for Google. Technically speaking I believe you are missing some blank spaces.

alahamdan




msg:3800801
 1:29 pm on Dec 5, 2008 (gmt 0)

Thanx for replay

what do you mean by im missing some blank spaced please?

Receptional Andy




msg:3800818
 1:59 pm on Dec 5, 2008 (gmt 0)

You should have a blank line for your second user-agent line, and I believe one right at the end.

Also, why the wildcard in the user-agent? I don't believe this is necessary. You can also drop the second of these two lines, since robots.txt uses prefix matching:

Disallow: /forum/style.php
Disallow: /forum/style.php*

alahamdan




msg:3800820
 2:02 pm on Dec 5, 2008 (gmt 0)

Yes i got your idea i will fix it now

Thanks very much for your help

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved