Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Googlebot Hitting /robots.txt Thousands of Times

Sometimes 4 or 5 times per second.

         

joeuz

12:36 am on Jul 23, 2005 (gmt 0)

10+ Year Member



Hello,

For the past few days, I have noticed on one of my websites that Googlebot is reading the robots.txt thousandsof consecutive times, sometimes 4 or 5 times every seconds. The IP is 66.249.65.139.

Does anybody know why this is happening?

Thanks

Joe

g1smd

11:34 am on Jul 27, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



You can email them and they will usually turn the wick down for you from their end.

I am not sure if there is some sort of Google-specific instruction that you can add to the robots.txt file to do this automatically.