homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

Can GoogleBot Spider GZipped Contents?

 5:45 pm on Aug 2, 2004 (gmt 0)

I have no idea what this question means... but one of my clients asked me this >>

Can Googlebot handle gzipped contents (on the fly compressed content a la mod_gzip)?

Does anyone have a clue as to what this is and if GoogleBot can handle this?



 7:49 pm on Aug 2, 2004 (gmt 0)

Googlebot does read the pages so no worries about activating gzip, but it reads the uncompressed page.

Mod_gzip will show the uncompressed page whenever the client who is visiting the page does not give a 'Accept-Encoding: gzip' header and Googlebot probably does not do that since I verified in my own logfiles it reads the uncompressed version. (ie. 31057 bytes against 4856 bytes compressed).

on a sidenote: inktomisearch does read the compressed pages (Yahoo Slurp bot)


 8:58 pm on Aug 2, 2004 (gmt 0)

Thank you for the response janjaap. Very much appreciated.


 11:41 pm on Aug 2, 2004 (gmt 0)

GG often talked about it in the past and said that he can't see a reason not to do it. Now knowing that Y! does it I wonder why G, having tens of thousands of servers, can't do it. Or if there's a reason against it.


 2:19 am on Aug 3, 2004 (gmt 0)

I look forward to the day when Googlebot saves us a bit of bandwidth by requesting the Gzipped version of our pages.


 9:31 pm on Aug 6, 2004 (gmt 0)

Daniel Dulitz says that day is very soon.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved