Thanks for the replies ;)
Actually they're all my sites that I run since 1996, we rented a server farm and moved the sites to several servers. There's plenty of work there, and over (way way way over) a million pages.
We do get a heap of google traffic as our subjects range from needles to satellites (both fictitious, we don't do needles OR satellites). MOST our pages don't sell a thing. No I'm not Slashdot.
My suggestion is not just for my own good, it's for the overall well being of the web. Our revenues from selling our widgets pay for the bandwidth (there's even a little left for friday beer ;))
But after I installed mod_gzip on my server farm (most browsers support gzip) the page size went from 20k average to 2k (amazing huh!) - but googlebot transfers remain in the 20k average and I've tcpdumped Googlebot requests - they indeed do not do Accept-Encoding: gzip.
If Googlebot used gzip it would cut Googlebot's bandwidth spending on my site from U$ 500.00 to U$ 50.00 monthly! From 6,000.00 yearly to U$ 600.00 yearly, just from adding GZIP support - painless, easy, quick!
Imagine what this would do for Google themselves, they pay for bandwidth too!
How could Google benefit from compression?
- Google would cut bandwidth costs
- If enough servers installed compression they'd spider the web much faster (if 100% of worldwide apache users installed mod_gzip I'd guess google could spider the web about 7 times faster, no kidding!)
- Compression is already used in images and video, why not on text? In case you're not a file format expert JPEG, MPEG, GIF, MP3, PNG and friends are just names of compression standards combined with a file format!
- Compression is standard and always accepted in computer science, it causes no information loss and gzip is very fast on Linux, which Google uses on cheap hardware
- Hardware is cheaper than bandwidth
How would Google stimulate enough webmasters to install gzip?
Instead of just terrorizing SEO's with their webmasters pages Google should add something like :
WEBMASTERS IF YOU CONFIGURE YOUR SERVER MAKE SURE TO ENABLE GZIP COMPRESSION ON IT - YOU'LL GET CRAWLED FASTER AND SAVE TONS OF DOLLARS AND MAKE THE WORLD A BETTER PLACE
Other engines would follow suit, the internet would be faster (about 10 times faster). Webmasters that didn't have gzip installed would install it just to have their pages load faster - believe me mod_gzip made my sites load about 5x times faster on AVERAGE - sometimes even reaching 10 times speed increase.
Microsoft showed extremely good attitude adding gzip support to all internet explorer software - therefore the 99% of my visitors that use internet explorer and mozilla already see my pages faster and they load about 7 times faster WITHOUT CHANGING A THING ON MY CONTENT.
Googleguy, as if you didn't get enough free consulting on this great forum here's a post I'd really appreciate if you read. Help make the Internet a better place and make Googlebot gzip-compliant, in C it's one library you gotta link to, in Python it is a module you load, in Perl it is a 1 minute change to your code.....don't know what Googlebot is written in (I'd bet C/C++ based on Altavista's Scooter) but it shouldn't take more than 1 day to change this!