fatpeter - 6:53 am on Jul 12, 2011 (gmt 0)

A while ago I had the following directive in robots.txt

User-Agent: *
Disallow: /cgi-bin/

but i had a problem with adsense not showing adverts on pages below the cgi-bin

for example cgi-bin/links/showpicture.cgi?ID=14063

I didn't want any content on the site under the cgi-bin indexed as it is all dupe content and the previous directive seemed to work just fine.

I changed the directive to
User-Agent: *
Disallow: /cgi-bin/

User-Agent: MediaPartners-Google
allow: /cgi-bin/

to allow adsense bot

Now google has started to index 80,000 pages under the cgi-bin.

Is my directive wrong ? I've searched and searched but i can't find a reason why they are indexing these pages...