zooros - 9:09 pm on Dec 25, 2003 (gmt 0)

another piece of information that you might find useful:

action: i have deactivated/rerouted most of my mirror domains and i have banned altavista from my multimedia folder via robots.txt.

result: the bandwith consumed from altavistas multimedia
crawlers has been greatly reduced (from several gigabytes
to a few hundred megabytes)

conclusion: 1 - altavistas crawler are NOT capable to identify multiple instances of the same file BEFORE they download the entire thing (not sure if they do it afterwards either ...) - this could be a loophole to get multimedia content into altavista
2 altavistas crawler do not follow the robots.txt protocol - or it takes some time before they reread the file - another flaw in altavista's technology
3 i wrote to them two weeks ago (corporate marketing) and i am still awaiting an answer - their customer relationship management is not very responsive ...