fatpeter - 6:53 am on Jul 12, 2011 (gmt 0)
A while ago I had the following directive in robots.txt
but i had a problem with adsense not showing adverts on pages below the cgi-bin
for example cgi-bin/links/showpicture.cgi?ID=14063
I didn't want any content on the site under the cgi-bin indexed as it is all dupe content and the previous directive seemed to work just fine.
I changed the directive to
to allow adsense bot
Now google has started to index 80,000 pages under the cgi-bin.
Is my directive wrong ? I've searched and searched but i can't find a reason why they are indexing these pages...