Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: mack
I think they should give up on building their search engine, it is too late.
[edited by: SEOPTI at 5:38 pm (utc) on Sep. 26, 2007]
That said, it makes such a poor job of crawling the site (despite a few sitemap files) that it is hard to say if it would ignore all of the robots exclusions if it ever worked properly.
Also discovered that it does not appear to understand a sitemap index file. Only when you explicitly put all the sitemaps in the robots file does the silly bot retrieve them.
Yes, it's cloaking, but with no intent to deceive anyone.