mbennie - 4:40 am on Mar 19, 2004 (gmt 0)

My sources tell me that sites in the Yahoo! Directory have always been crawled.

My site does not get crawled. It has been in both the Yahoo! directory and DMOZ for more than 2 years. Tim has already confirmed that there is an old Inktomi ban on the site. Nobody seems to want to tell me what the problem is.

Every day slurp comes by and gets the robots.txt 6 or 7 times. It doesn't get any of the pages.

The site has been successfully crawled in its entirety by Google, FAST and Alta Vista as all pages are straight .htm extensions.

So, IMHO, being in the directory does not guarantee crawl or inclusion in the index.