Oct 10-11 massive crawling - anyone?

bots hitting hard

[httpwebwitch]

chime in: has anyone noticed insane amounts of bot traffic in the past 24 hours? Or any other unusually agressive behaviour?

[tedster]

A bit heavy, but nothing out of normal bounds here. What have you seen? How far out of the ordinary is it?

[selomelo]

As far as my sites concerned, google was inactive for the last couple of days hitting a few pages daily. Today, I see an increase in number of pages hit. Nevertheless, it is not extraordinary. In fact, it is far below the average.

[httpwebwitch]

OK - just sending out feelers, to see if something big was happening outside.

EOM

[Car_Guy]

In the first four days of this month, my site's robots.txt file has been accessed 888 times, for an average of 222 times a day.

[Simsi]

OCT 10-11 chime in: has anyone noticed insane amounts of bot traffic in the past 24 hours? Or any other unusually agressive behaviour?

Sorry we're clearly in different time zones. Gimme 5 days to catch up and I'll let you know :-)

(but up to now, no not seen massive crawling. It's still only the 5th Oct here mind!)

[nippi]

getting totally hammered by mozilla google

[steveb]

I guess I've been in a coma for a week.

[trinorthlighting]

I have been getting hammered today as well!

[Patrick Taylor]

Chiming in: nothing different.

[tedster]

People who are seeing intense crawling -- by any chance have you had (in the recent past - you can't tell today) excessive supplemental results? I'm wondering if this is another refresh of the Supplemental Index. The currently reported "broken" condition of the site: operator search makes me wonder if there's a connection.

[trinorthlighting]

We have quite a few supplementals (Repair parts) and we have also been adding products for the Christmas season. I will check our supplemental cache dates. It would make sense since the pages that have been crawled really hard are our supplemental pages.

[trinorthlighting]

Here is a strange one for you, the preffered domain kicked in on one of our sites (It does not have 301's in place) and all the pages that used to be in the regular index went supplemental. Anyone else noticing that? For some reason our google traffic is way up though....

[ltedesco]

Oct 10-11 massive crawling - anyone?

Oh man! Where are you located?

[twebdonny]

I sure wish we could get our supps recrawled

[ann]

Actually page rank was being updated today.

Ann

[twebdonny]

It's the Florida Hurricane Recovery Effort....

[walkman]

I have this feeling that Google will do a more comprehensive update before the holiday. Why?
Cause I said so! ;)

First, we're due for a real update.
Second, timing is perfect: not too close to Christmas not too far.

[plasma]

Actually page rank was being updated today.

What DC? Are you sure, that it's not only the last week's PR update?

[Whitey]

Massive bot activity in last 24hrs [ Google ]
Site:tool - not working properly
Site:tool - results showing mixture of old and new data ie old meta title , current description , correct URL or v.v.
Site:tool - Disparity yesterday between regional results [ looked perfect ] and .COM [ wild and wrong ]
Site:tool - shows screwy "supplementals" returning after they were fixed [ i thought ]
PR toolbar - lowered on one site yesterday
PR others - steady
No change to traffic
Still showing filtered results

[edited by: Whitey at 8:53 am (utc) on Oct. 6, 2006]

[mcskoufis]

Massive googlebot activity since this morning, although yahoo seems to be crawling heavy as well on all my and my customers' sites...