What is wrong with my Robots.txt file?

I'd appreciate help with my Robots.txt file

[Morthan]

Let me start by saying that I am clearly less experienced than most of you webmasters. Rrecently my traffic has doubled but my AdSense revenue is now about 50% of what it used to be only a few months ago.

But here is the actual robots.txt file

User-agent: *
Disallow: /awstats/
Disallow: /book_pictures/
Disallow: /cgi-bin/
Disallow: /chrisoconnor/
Disallow: /cp/
Disallow: /emoticons/
Disallow: /ezboard/
Disallow: /ian/
Disallow: /images/
Disallow: /johnoconnor/
Disallow: /junk/
Disallow: /member%20photos/
Disallow: /member_photos/
Disallow: /modlogan/
Disallow: /posticons/
Disallow: /webalizer/
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /templates/
Disallow: /under_cons.php
Disallow: /TEMPLATE.php
Disallow: /supportold.php
Disallow: /quote.php
Disallow: /quote_view.php
Disallow: /thanks.php
Disallow: /memberintros.php
Disallow: /member_photos.php
Disallow: /common.php
Disallow: /groupcp.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /posting.php
Disallow: /profile.php
Disallow: /privmsg.php
Disallow: /viewonline.php
Disallow: /faq.php
Disallow: /updates-topic
Disallow: /stop-updates-topic
Disallow: /ptopic
Disallow: /ntopic
Disallow: /post-

When I use the Google Webmasters tool for analyzing the Robots.txt file I get a "Status 200 (Success)" which I assume means my file is correctly written.

But when in Google AdSense I click on the "Site Diagnostics" and I get a completely different message, which is that both of the following URL's are "blocked" due to my "Robots.txt" file.

http://example.com/
http://www.example.com/

How can I pass the Robots.txt. analyzer tool yet have my URL blocked? Any ideas?

Google explains:

"If you have a robots.txt file, the page requesting Google ads may be marked as 'disallow' in your robots.txt file. You can update your robots.txt file to give us access by following these instructions.
If you do not have a robots.txt file, please create one in the root directory of your domain and then update it by following these instructions.

If you don't know whether you have a robots.txt file, please consult your web provider or webhost and ask them to modify the robots.txt file on your domain to comply with these instructions. "

Again, go back to my actual Robots.txt file. I don't want any robots indexing those pages. But there are Google ads on them in the sidebars. Both of the sidebars are created as includes and they exist on all pages, not just the ones that I want indexed. Could this be the problem? Is the actual design of my site the problem? Am I seeing a reduction in my AdSense revenue because a good percentage of the pages that have the Google AdSense code are disallowed by the Robots.txt file?

If this is the problem I would have to see if my web designer can fix this. Any help would be appreciated. Or maybe it doesn't matter that I am getting that Blocked URL message on the Site Diagnostics screen. Or does it?

Argghh

Thanks for your help...if you decide to held.

Chris

[edited by: engine at 12:01 pm (utc) on Feb. 22, 2008]
[edit reason] examplified and specifics removed [/edit]

[RonS]

I don't know why they think they are blocked, but you could try adding:

User-agent: Mediapartners-Google*
Disallow:

To the top of your robots.txt file to allow the AdSense bots to scan your pages.

[edited by: RonS at 10:41 pm (utc) on Feb. 21, 2008]

[Morthan]

Thank you for the quick reply, RonS. :)

Would that text simply sit on top of the rest of my Robots.txt file? I am a novice at this stuff and I'd hate to make an error that results in my site no longer being indexed.

So should it look like this...

User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: /awstats/
Disallow: /book_pictures/
Disallow: /cgi-bin/
Disallow: /chrisoconnor/
Disallow: /cp/
Disallow: /emoticons/
Disallow: /ezboard/
Disallow: /ian/
Disallow: /images/
Disallow: /johnoconnor/
Disallow: /junk/
Disallow: /member%20photos/
Disallow: /member_photos/
Disallow: /modlogan/
Disallow: /posticons/
Disallow: /webalizer/
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /templates/
Disallow: /under_cons.php
Disallow: /TEMPLATE.php
Disallow: /supportold.php
Disallow: /quote.php
Disallow: /quote_view.php
Disallow: /thanks.php
Disallow: /memberintros.php
Disallow: /member_photos.php
Disallow: /common.php
Disallow: /groupcp.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /posting.php
Disallow: /profile.php
Disallow: /privmsg.php
Disallow: /viewonline.php
Disallow: /faq.php
Disallow: /updates-topic
Disallow: /stop-updates-topic
Disallow: /ptopic
Disallow: /ntopic
Disallow: /post-

[RonS]

I'd put a blank line between the two sections, eg:

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /awstats/
[...snip...]

but other than that yes, give it a whirl.

[edited by: RonS at 10:44 pm (utc) on Feb. 21, 2008]

[tim222]

My robots.txt doesn't look very much different than yours, with the exception of a lot less entries. However the format is pretty much the same. I'm getting a message that some pages are blocked because of robots.txt, but not the whole domain like you are.

Here's my robots.txt:

# robots.txt
User-agent: *
Disallow: /old_site_images/
Disallow: /images/
Disallow: /css/
Disallow: /contact.php
Sitemap: http://www.example.com/sitemap.xml

[edited by: engine at 12:02 pm (utc) on Feb. 22, 2008]
[edit reason] examplified [/edit]

[Morthan]

I'm going to give your idea a shot, RonS. I'll post back here letting you know if the problem has been fixed. I appreciate the help guys.

[FromRocky]

Morthan,
Why don't you run a test to test these "URLs against this robots.txt file"?

You can test against both Googlebot and Mediapartners-Google.