Googlebot and gzip

[BartVB]

Is there any news on:

[webmasterworld.com...]

This is about sending gzipped pages to Google.. At the moment my site is getting indexed by Google. I found out about that while looking at my traffic stats. Normally my site does approximately 90GB per month but now (may 21th) we are already at 120GB, mainly because of Google :\ Of course it's great that it's indexing the site but it's indexing a LOT of 50-100k pages that can be 4-20k.. Quite a waste of bandwidth and money (1GB costs approx 2 to 3 dollars in The Netherlands).

It would be extremely convenient for me (and IMO for Google too) if GoogleBot could speak gzippish.

[edited by: heini at 7:17 pm (utc) on May 21, 2003]
[edit reason] no urls please / thank you! [/edit]

[GoogleGuy]

I think someone here did a test and was surprised at how many web servers don't serve compressed pages correctly (like the early days of virtual hosting). So I wouldn't expect this anytime soon, but I agree in principle that this would be a good feature in the future.

[ruserious]

Proper caching (last modified/ETAG) would reduce the amount of traffic from Google, although it might be a marginal gain, if you have very large AND very active forums.
But it is also a feature I'd like to see in the future version of phpBB. :D But I guess I am getting offtopic... *runs for cover* ;)

[shrirch]

You may also save some bandwidth by using robots.txt.

If you're a forum site, make sure "newthread" "newreply" type links are not followed. Saved me about 2GB of bandwidth over the last month with google. Add gif's and jpgs to the list of disallowed files.

Also, be very careful of Alexa. If you think google is bad, wait till that beast figures it needs to archive your sites every couple of weeks.