Msg#: 3236239 posted 7:17 pm on Jan 29, 2007 (gmt 0)
Now I know this must be a dream come true for some people, but I've got a forum I've recently started, couple hundred of members and posts, and it would seem the google spider is on it practically 24/7, and I have no idea why! I dont care about the bandwidth, but it does seem rather odd. Theres around 20gb of files in a download section, and it seems to spend a bit of time in there... it's not downloading the files as far as I can tell, so any ideas as to what it's actually doing? This has been going on for days now.
Msg#: 3236239 posted 8:14 pm on Jan 29, 2007 (gmt 0)
This is the common--though little spoken of--problem with dynamic sites. Your pages either do not provide a Modified server-header, or provide a Modified server-header with the current date, and/or do not respond to an If-Modified-Since request-header.
The above means that SEs such as Google see your pages to be constantly updated, and therefore hammer your site.
Provide the required headers and this 'problem' will go away.