dusky - 11:53 pm on May 8, 2010 (gmt 0)
On the adsense page under Site Authentication, you give access details to G* and their bot uses that as a signed member to access content for adsense purposes!
G* should have an equivalent on WMT, but I guess there are security and privacy implecations if they did.
My understanding is you either open for all or close for all. If you want the pages indexed you are also asking everyone to read them.
One way of making things better is to allow only the first post or even better, the start of the thread to non-signed members, that way those pages get indexed, people will come and read the first post BUT are asked to login if they want to read the rest. Should not be too difficult to hack your perl subs, php functions or asp code to do it. This will get you every thread indexed regardless on how many posts it has anyway, incidentally, in most cases Gbot only collects the thread page and ignores the page 2, page3, page 4 etc because on most forum software those pages are treated as one page unless you are clever enough to differentiate each page with meta titles and descriptions for each, implementing what I said is as good as any from that perspective.