jmccormac - 7:51 am on Jun 30, 2004 (gmt 0)

Does the msn bot/search service respect a robots.txt ban? Specifically, if we say tha directoryX is banned in robots.txt, will it attempt and download files in directoryX?

Based purely on msnbot's actions on one of my main directory sites, msnbot downloads banned directories. It also ignores HTTP results codes and was indexing the site every five days without bothering to use 304 results to indicate that pages had not changed. Considering that hostmasters pay for their bandwidth, letting an incompetently designed spider like msnbot loose on their site is a bad thing.

Regards...jmcc