santapaws - 5:48 pm on Sep 23, 2012 (gmt 0)

no its not a question of not browsing its a question of how to treat what are clearly low grade pages to google when being spidered. Doing nothing means it gets an a directory listing which google themselves index as an index html page. Clearly bad. if i turn off directory browsing google gets 100's of not authorized returns, so it doesnt know what is being forbidden. If i add an index page and put no-index in the meta then it sees 100's of no-index pages which will be about 10% of the site, wonder if thats a problem given the thoughts that such pages if a significant amount MAY hurt the site overall.