Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
joined:Apr 13, 2001
They could at least have the decency to take them and serve them from their own machines like google does don't you think.
So is this new or has alltheweb.com been serving images for awhile? This is my first experience with this.
Google is very polite and will respect this in robots.txt:
I see that Fast says it respects robots.txt (user-agent "fast"), but does anyone know if it will respect some kind of wildcard designation for images, as Google does?
(Yes, I know, put them all in one directory.... change the names.... etc.) :-(