Forum Moderators: Robert Charlton & goodroi
Several months later, the cgi-bin was still showing PageRank, and a search for the URL indicated that, indeed, the URL was still in Google's index.
Since then, I instructed our engineer to additionally employ .htaccess and meta noindex to make it very clear to Google that we do not want the cgi-bin spidered.
After the recent Google PR update, I noticed:
1. Areas of my sites that had ugly query strings that were addressed with mod rewrites now had PR
2. The cgi bin still had PR and was in the index!
I tried to check the last time the googlebot had spidered the cgi-bin, but was dismayed to realize our log files had reached their 2 gig limit months ago and were not flushed by our lazy engineers, so I cannot say WHEN Google last visited. It could be possible the bot hasn't visited in ages and the cgi-bin is just a historic artifact in the index.
Any ideas?