homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google Desktop Tools and Google Labs Projects
Forum Library, Charter, Moderator: open

Google Desktop Tools and Google Labs Projects Forum

Googlebot on my site 24/7
Google bot on site constantly

 7:17 pm on Jan 29, 2007 (gmt 0)

Now I know this must be a dream come true for some people, but I've got a forum I've recently started, couple hundred of members and posts, and it would seem the google spider is on it practically 24/7, and I have no idea why! I dont care about the bandwidth, but it does seem rather odd. Theres around 20gb of files in a download section, and it seems to spend a bit of time in there... it's not downloading the files as far as I can tell, so any ideas as to what it's actually doing? This has been going on for days now.




 8:14 pm on Jan 29, 2007 (gmt 0)

This is the common--though little spoken of--problem with dynamic sites. Your pages either do not provide a Modified server-header, or provide a Modified server-header with the current date, and/or do not respond to an If-Modified-Since request-header.

The above means that SEs such as Google see your pages to be constantly updated, and therefore hammer your site.

Provide the required headers and this 'problem' will go away.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google Desktop Tools and Google Labs Projects
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved