g1smd - 8:04 pm on Aug 13, 2006 (gmt 0)

OK. I follow the logic, but I've read robotstxt.org many times, and never got the hint that it worked that way before.

I can't believe I have missed this important fact for 8 years, but this is the first time I have added a specific section for Googlebot to a robots.txt file. I much prefer using the robots meta tag on each applicable page.

We need to make sure the "new" information is widely disseminated, as I expect that many others have missed it too.