caine - 8:46 am on Aug 11, 2003 (gmt 0)

Obviously i don't know what your site is about, hence whether the 20k pages are built around a products database, which would have 'manufacturer' 'type' 'model no#' and other characteristics, then i would suggest - yes - especially if the content on the pages is unique to each other.

If however the conent is not for commercial advertising, then the cost of not having some kind of income from the site, may outweigh the cost of uping your upstream throughput.

Unique commercial content - as long as the site is built well -> worth allowing all bots to crawl.
If not commercial - then i would look at restrictive measures, on pages that probably don't need to be crawled.