Dreamquick - 11:45 am on Aug 16, 2002 (gmt 0)

I have trouble believing any reputable/large search engine would ignore robots.txt - if this did happen that spider would find itself being physically blocked from the majority of managed websites sooner or later.

It is not unreasonable to suggest that if a search engine spider can't gain access to lots of websites this would lead to a less useful search engine, once they are in this situation then there are only so many options;

1) Fix your search engine to work with robots.txt
2) Leave the search engine business
3) Carry on and pretend that everything is fine

Obvious it's a lot easier to build a working spider (or at least learn from webmaster comments that describe where your spider is failing) than it would be to only fix your spider when lots of sites have blocked it and your business is failing as a result.

People are generally very tolerant of most things SE spiders do - this does not include ignoring robots.txt as this is a very clear cut thing as it protects both the website and the spider.

- Tony