nileshkurhade - 12:21 pm on Jan 13, 2004 (gmt 0)

Seems the "Oxford English Dictionary" site [oed.com...] is also suffering the same fate, even with its PR of 9. Only shows 43 pages and almost all pages left only have the URL and Similiar Pages link.

The robots.txt of [oed.com...] looks like below, is it the reason why there are only 43 pages indexed.

User-agent: *
Disallow: /accesslogs/
Disallow: /ads/
Disallow: /apps/
Disallow: /backtocs/
Disallow: /browse/
Disallow: /cgi/
Disallow: /conf/
Disallow: /content/
Disallow: /content
Disallow: /feature/
Disallow: /future/
Disallow: /guides/
Disallow: /home/
Disallow: /help/
Disallow: /icons/
Disallow: /include/
Disallow: /math/
Disallow: /subscribe/
Disallow: /news/
Disallow: /about/
Disallow: /services/
Disallow: /framesok/
Disallow: /general/
Disallow: /pdfs/
Disallow: /readers/
Disallow: /tour/
Disallow: /archive/