Googlebot consumes my bandwidth!

[Hanuman07]

I have something that starts to look like a problem. I run a small forum with something like 10-20 topics per day. We have about 5,000 topics in total. Google eat almost 80% of my bandwidth meaning about 6Gb per month.

I am using a robots.txt file but it does not seem to do the trick. I want Google to spider but not the whole site every hour. Anyone here who have solved this problem?

I am using Googles webmastertool, but changing the speed which googlebot spider makes no difference.

What I would like to do is to ask Googlebot to visit only one time per day...

The site is < no personal domains, please >

[edited by: tedster at 2:21 pm (utc) on Oct. 11, 2007]

[Gene_B]

I just found Google using a much larger amount of bandwidth in the last week then they ever used. I also found people linking in from Google to pages I don't have! I found a directory had chmod to 777 and a .htaccess file I never had which pointed the 404 errors to a php file which somehow delivered all kinds of things.

I had to put in a mod-rewrite to deliver a 410 "Gone" result after I removed stuff from the folder.

[edited by: tedster at 5:14 pm (utc) on Oct. 11, 2007]