msndude - 4:36 am on Jun 22, 2006 (gmt 0)

RichTC is correct. It's possible for us to fail to crawl your site as deeply as we ought to, even though you didn't make big mistakes in your HTML. This is another problem we're actively working on.

Don't let me stop you from doing that HTML cleanup if you really want to do it, but please don't spend hours and hours on it hoping that'll change how we index your site. It almost certainly won't.

Now if you screw up your robots.txt file, THEN we may well fail to crawl your site, but if that happens, we won't crawl it at all.