Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Googlebot on my site 24/7

Google bot on site constantly



7:17 pm on Jan 29, 2007 (gmt 0)

5+ Year Member

Now I know this must be a dream come true for some people, but I've got a forum I've recently started, couple hundred of members and posts, and it would seem the google spider is on it practically 24/7, and I have no idea why! I dont care about the bandwidth, but it does seem rather odd. Theres around 20gb of files in a download section, and it seems to spend a bit of time in there... it's not downloading the files as far as I can tell, so any ideas as to what it's actually doing? This has been going on for days now.



8:14 pm on Jan 29, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

This is the common--though little spoken of--problem with dynamic sites. Your pages either do not provide a Modified server-header, or provide a Modified server-header with the current date, and/or do not respond to an If-Modified-Since request-header.

The above means that SEs such as Google see your pages to be constantly updated, and therefore hammer your site.

Provide the required headers and this 'problem' will go away.


Featured Threads

Hot Threads This Week

Hot Threads This Month