homepage Welcome to WebmasterWorld Guest from 54.204.68.109
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Google / Google Desktop Tools and Google Labs Projects
Forum Library, Charter, Moderator: open

Google Desktop Tools and Google Labs Projects Forum

    
Googlebot on my site 24/7
Google bot on site constantly
nickakam




msg:3236241
 7:17 pm on Jan 29, 2007 (gmt 0)

Now I know this must be a dream come true for some people, but I've got a forum I've recently started, couple hundred of members and posts, and it would seem the google spider is on it practically 24/7, and I have no idea why! I dont care about the bandwidth, but it does seem rather odd. Theres around 20gb of files in a download section, and it seems to spend a bit of time in there... it's not downloading the files as far as I can tell, so any ideas as to what it's actually doing? This has been going on for days now.

Rgds,
Nick

 

AlexK




msg:3236312
 8:14 pm on Jan 29, 2007 (gmt 0)

This is the common--though little spoken of--problem with dynamic sites. Your pages either do not provide a Modified server-header, or provide a Modified server-header with the current date, and/or do not respond to an If-Modified-Since request-header.

The above means that SEs such as Google see your pages to be constantly updated, and therefore hammer your site.

Provide the required headers and this 'problem' will go away.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google Desktop Tools and Google Labs Projects
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved