Googlebot - phpbb - session id's - (deprecated) Google News Archive forum at WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

Googlebot - phpbb - session id's

How can I stop googlebot from bringing down my site?

Swamper

4:02 pm on May 30, 2003 (gmt 0)

As the description says, how can I limit session id's so Googlebot doesn't keep loading my server. Googlebot crashed my server today by putting such a high load on my MySql database.

I have tried a fix for phpbb to not give http_useragent googlebot a session id.

I have tried to use robots.txt to tell googlebot to stop indexing the pages
Disallow: modules.php?name=Forums

(this is a phpbb module for phpnuke)

The only thing I haven't been able to do is to see if there is something I can add into my htaccess file that would prevent googlebot from grabbing session id's.

Does anyone have any ideas if this is possible with an htaccess file?

Thanks

trillianjedi

4:08 pm on May 30, 2003 (gmt 0)

Swamper,

I think you have actually fixed the problem.

The problem is google still has the old SID URL's in it's data.

Check your logs - I think you'll find that the SID's google is coming in on are old ones.

It may take a while to flush through. Google probably also has an old cache of your robots.txt.

You may need to mail google again I'm afraid.

Swamper

4:15 pm on May 30, 2003 (gmt 0)

I didn't want to have to email them, but I have now.
I will see what their response is.

Thanks

trillianjedi

4:17 pm on May 30, 2003 (gmt 0)

It always helps to mail them an extract of your logfile too - they can match up the SID's....

Swamper

12:30 pm on Jun 1, 2003 (gmt 0)

Mailing them all of the session id's would take hours.
There are thousands of hits with different id's.

As much as I didn't want too, I took the site down for two days while googlebot finished checking with the session id's. It might hurt my rating but I had to do it. My ISP got a little ticked when it brought down the server with approx. 20 domains running on it. The mysql load was just too much for it to take. One thing I probably have going for me is that I have a 404 redirect set up with all of my meta tags. It might save me (I hope)!

So far my rating hasn't dropped.

panos

1:12 pm on Jun 1, 2003 (gmt 0)

As the description says, how can I limit session id's so Googlebot doesn't keep loading my server. Googlebot crashed my server today by putting such a high load on my MySql database.

i had the same problem .
i fixed it using htaccess and dropping the sid for googlebot
also you may need to make some changes at the sessions table because often spiders are getting a new sid everytime they visit the forums

take a look at these threads :

[phpbb.com...]