Hey I heard there is a way you can delay a crawler on your site. Right now Google is hammering my site and has been for the past month. I find it very weird because it keeps searching the same pages over and over.
Does anyone know if I can delay the crawler so it doesn't go so fast?
I just got a reply back from Google. They said the Googlebot was having trouble because I had php session ids showing in the url. It was causing problems. They provided a link on how to remove it which really helped because I have been wanting to remove it for a while and I didn't know how. It was a problem relating to my PHPBB forums.