Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
I need Google Service for Organic Results and Traffics but at the same time I need to control the spider request. I tried Webmaster Tool >> Tool >> Set Crawl Rate >> Slow option, but there is no effectÖ
Kindly let me know if it is any other way to control GoogleBot?
Are you sure it is Google? Some people change their browser's user agent to googlebot. This is also a common practice for data miners. Be careful not to just look at user agents but to confirm the different ips are actually owned by Google.
Also Google has multiple bots. They have one for search, adsense, etc. If you are participating in multiple google programs you should expect these bots to be visiting your pages. This will drive up the amount of bot traffic on your site.
I am very sure, it is GoogleBot spider only, and All the IPs is owned by Google Inc.
I have blocked GoogleBot IP then swiftly a different GoogleBot IP is crawl my site, and if I removed from the blocking list after that very immediately Google Spider happening on my site, I know the reason because itís well designed Link Structure (Not Spam), is there any other way to control?