Forum Moderators: open
They could at least have the decency to take them and serve them from their own machines like google does don't you think.
So is this new or has alltheweb.com been serving images for awhile? This is my first experience with this.
Google is very polite and will respect this in robots.txt:
User-Agent: Googlebot
Disallow: /*.gif$
Disallow: /*.jpg$
I see that Fast says it respects robots.txt (user-agent "fast"), but does anyone know if it will respect some kind of wildcard designation for images, as Google does?
(Yes, I know, put them all in one directory.... change the names.... etc.) :-(