Brett_Tabke - 8:30 am on Apr 16, 2001 (gmt 0)
One thing I noticed while doing the robots.txt crawl/validator over at SEW, was so many people were redirecting to a 404. Often that 404 was "seamless" with no redirect or forward of any nature. You just pull the robots.txt and get their 404 page. A search engine will then have to figure out if it is really a robots.txt or an html page. That is pretty easy to do (look for <html> or <body> tags), but if it is a frame set page, it can be confusing. One trick I did find, was someone using a frameset page that actually worked as both a robots.txt AND as the frameset page. This knowledge will be kept under lock and key - it's the last thing we need on the web.