google bot taking 1600 MB per day

66.249.72.72 massive crawling

[Jaunty Edward]

Hi,

my site is huge(400,000+ content pages) and its a little over a year old.

we used to do around 1000 page views a day with around 7-8 GB BW every month.

From last month a google bot 66.249.72.72 has been crawling the site very fast. Last month our BW was 44.6 GB and this month its already 9.8 GB in just 5 days. Whenever I check the latest visitors in the cpanel there is this huge list of entries from this IP.

The traffic too has gone up from 800 to 2200 page views a day.

I dont mind if the bot crawls at even 15 times faster speed... as long as its google. I dont want to reduce the speed or control the bot, just want to make sure its google and everything is ok.

thanks
bye

I am wondering if there is anyone else whoz site is being crawled at this speed.

[tedster]

Matt Cutts has given us the details on how to verify that a bot is really googlebot.

[googlewebmastercentral.blogspot.com...]

[mjtkop]

Hi, I also think this bot from the same IP is using a lot of my bandwidth to and it looks like it is doing some other scary stuff on my site, let me explain;

I use a CMS with a security module installed that can stop sql injection attempts, hihg loading crawlers and other attacks, for the past couple of days this IP has been logged as attempting to do some sort of issolated comment attack, not sure what an issolated comment attack is exactly but it cant be good.

I did a reverse dns and the IP/bot is coming from googlebot.

Im not sure if i should ban this IP or not I want google to be able to crawl my site, any ideas on what could be happening?

[DamonHD]

Quite sure its not coming from the Google Web Accelerator (GWA), ie bad guys proxying their attack via Google?

Rgds

Damon

[goubarev]

Hey Jaunty Edward,

I have very similar site to yours ~ 1 year old, 300k pages.
I do too see increase in googlebot traffic - averaged 5,000 pages per day (~200Mb/day) in since begining of April.

It seems these numbers are high, but stil within the historic limits of what Google used to download for my site. I have similar spikes in the middle of Feb and at the end of Dec.

[goubarev]

Dude, hold on,
something doesn't add up...
You are saying 1.6Gb/day = 2200 pages
That's like 720Kb per page?
Your pages are HUGE?!

[Jaunty Edward]

goubarev,

I did not say my site is 2200 pages... its 2200 page views a day..... and I dont know how many pages are being scanned by google everyday ... thats causing 1.6 GB a day.

Thanks,
bye