jetteroheller - 12:19 pm on Dec 9, 2012 (gmt 0)
I have all my htm and js files Gzip compressed on the server.
My provider does not implement the "expand compressed file if client does not support gzip delivery".
So I implemented in .htaccess a redirect for not gzip capable clients.
Now I discovered this discussion, because nearly only amazonaws spiders are on the redirected pages.
There are 2 possibilities, that a client wants the uncompressed contendt:
1.) Can not gzip and is therefore redirected
2.) It happens by a linke from somebody with problem 1
To eleminate point 2, my system now redirects all to the compressed content if HTTP_ACCEPT_ENCODING contains gzip.
Seems AmazonAWS is the only spider not able to understand gzip.