WyrdoManx - 9:14 am on Oct 22, 2003 (gmt 0)

Hi Jim,

Thanks. I'm a long time lurker, first time poster. :)

The robots.txt size is... um, erm, ha... a mere 38,357 bytes. So that's about 37M. Did I mention it was huge?

It passes the validation tests. I found your post on how to shorten robots.txt extremely useful. This particular file is silly in length because it excludes so many different URLs explicitly.

I've never considered emailing a search engine (or the people who work there) directly before. I might just get over my "I'm not worthy!" shtick and give it ago. I've also told the mastermind behind this particular robots.txt that meta tagging the specific pages in question and using robots.txt to protect entire directories might be the way to go in this case.