MarkHutch - 4:54 pm on Mar 11, 2002 (gmt 0)

Thanks for the replies. It sure seems like they are spending a bunch of time and bandwitdth trying to spider pages that are not there anymore. However, maybe they have their reasons for not checking more often for a robots.txt file. Maybe most people don't use one and they just don't want to waste their time checking too often or something like that...