Forum Moderators: open
"If Google and Yahoo don't want that data, they can not index it," (the AnSearch CEO) told ZDNet Australia. "It's not up to us to tell the other people in our space what they can and can't index."
[edited by: bill at 7:41 am (utc) on Oct. 19, 2005]
[edit reason] fixed link [/edit]
the next version of Ansearch's 'spider' software .... would ignore Web sites which explicitly opted out of general search engine indexing through their public configuration files.
So sometime in the future they will obey robots.txt - but are incapable of using it themselves.
They obviously have no clue.