Welcome to WebmasterWorld Guest from 54.147.212.98

Forum Moderators: open

Message Too Old, No Replies

Googlebot on my site 24/7

Google bot on site constantly

   
7:17 pm on Jan 29, 2007 (gmt 0)

5+ Year Member



Now I know this must be a dream come true for some people, but I've got a forum I've recently started, couple hundred of members and posts, and it would seem the google spider is on it practically 24/7, and I have no idea why! I dont care about the bandwidth, but it does seem rather odd. Theres around 20gb of files in a download section, and it seems to spend a bit of time in there... it's not downloading the files as far as I can tell, so any ideas as to what it's actually doing? This has been going on for days now.

Rgds,
Nick

8:14 pm on Jan 29, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This is the common--though little spoken of--problem with dynamic sites. Your pages either do not provide a Modified server-header, or provide a Modified server-header with the current date, and/or do not respond to an If-Modified-Since request-header.

The above means that SEs such as Google see your pages to be constantly updated, and therefore hammer your site.

Provide the required headers and this 'problem' will go away.