MarkHutch - 11:10 pm on Mar 18, 2002 (gmt 0) (edited by: MarkHutch at 6:55 pm (utc) on Mar. 26, 2002)
Hello Joel and thanks for explaining how your service works. None of our sites are part of any referral link program, but you still crawl our sites. I went to your website several months ago and found out that you do not honor robots.txt. Can any member of your service request a site be crawled, even if that site is not part of a referral program? If so, what type of information are you providing to your members about sites you crawl for members? Something about this just doesn't feel right to me, but maybe I'm just not seeing the full picture of what's going on.
(edited by: MarkHutch at 6:55 pm (utc) on Mar. 26, 2002)