WyrdoManx - 12:54 pm on Oct 21, 2003 (gmt 0)

Hiya,

I'm looking at a report that suggests FAST is being refused access to certain directories by the robots.txt file.

I'm looking at the robots.txt file.

I'm not looking at a robots.txt file that refuses FAST access to these directories, nor am I looking at an incorrectly formatted robots.txt.

I am, however, looking at a huge robots.txt file. It's not small. It's large. Very.

Could this be a problem? Do you think FAST might just cut off the robots.txt after X hundred or Y thousand bytes? A sudden cut might just annoyingly leave a trailing / in the command and block the entire directory that way.