Welcome to WebmasterWorld Guest from 54.166.211.248

Forum Moderators: Robert Charlton & andy langton & goodroi

Message Too Old, No Replies

Oct 10-11 massive crawling - anyone?

bots hitting hard

     
5:44 pm on Oct 5, 2006 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4061
votes: 0


chime in: has anyone noticed insane amounts of bot traffic in the past 24 hours? Or any other unusually agressive behaviour?
6:25 pm on Oct 5, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


A bit heavy, but nothing out of normal bounds here. What have you seen? How far out of the ordinary is it?
6:35 pm on Oct 5, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:July 22, 2005
posts:151
votes: 0


As far as my sites concerned, google was inactive for the last couple of days hitting a few pages daily. Today, I see an increase in number of pages hit. Nevertheless, it is not extraordinary. In fact, it is far below the average.
6:55 pm on Oct 5, 2006 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4061
votes: 0


OK - just sending out feelers, to see if something big was happening outside.

EOM

8:14 pm on Oct 5, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:June 10, 2006
posts:606
votes: 0


In the first four days of this month, my site's robots.txt file has been accessed 888 times, for an average of 222 times a day.
8:14 pm on Oct 5, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 13, 2005
posts:716
votes: 0


OCT 10-11 chime in: has anyone noticed insane amounts of bot traffic in the past 24 hours? Or any other unusually agressive behaviour?

Sorry we're clearly in different time zones. Gimme 5 days to catch up and I'll let you know :-)

(but up to now, no not seen massive crawling. It's still only the 5th Oct here mind!)

8:15 pm on Oct 5, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 3, 2003
posts:809
votes: 0


getting totally hammered by mozilla google
8:59 pm on Oct 5, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


I guess I've been in a coma for a week.
10:42 pm on Oct 5, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 5, 2006
posts:2095
votes: 2


I have been getting hammered today as well!
10:52 pm on Oct 5, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 30, 2003
posts:932
votes: 0


Chiming in: nothing different.
12:42 am on Oct 6, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


People who are seeing intense crawling -- by any chance have you had (in the recent past - you can't tell today) excessive supplemental results? I'm wondering if this is another refresh of the Supplemental Index. The currently reported "broken" condition of the site: operator search makes me wonder if there's a connection.
12:43 am on Oct 6, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 5, 2006
posts:2095
votes: 2


We have quite a few supplementals (Repair parts) and we have also been adding products for the Christmas season. I will check our supplemental cache dates. It would make sense since the pages that have been crawled really hard are our supplemental pages.
12:50 am on Oct 6, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 5, 2006
posts:2095
votes: 2


Here is a strange one for you, the preffered domain kicked in on one of our sites (It does not have 301's in place) and all the pages that used to be in the regular index went supplemental. Anyone else noticing that? For some reason our google traffic is way up though....
1:04 am on Oct 6, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 9, 2004
posts:77
votes: 0


Oct 10-11 massive crawling - anyone?

Oh man! Where are you located?

3:01 am on Oct 6, 2006 (gmt 0)

Preferred Member

joined:July 19, 2002
posts:415
votes: 0


I sure wish we could get our supps recrawled

ann

3:10 am on Oct 6, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 25, 2002
posts:2605
votes: 0


Actually page rank was being updated today.

Ann

3:14 am on Oct 6, 2006 (gmt 0)

Preferred Member

joined:July 19, 2002
posts:415
votes: 0


It's the Florida Hurricane Recovery Effort....
4:09 am on Oct 6, 2006 (gmt 0)

Senior Member

joined:Dec 29, 2003
posts:5428
votes: 0


I have this feeling that Google will do a more comprehensive update before the holiday. Why?
Cause I said so! ;)

First, we're due for a real update.
Second, timing is perfect: not too close to Christmas not too far.

8:37 am on Oct 6, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 13, 2003
posts:630
votes: 0


Actually page rank was being updated today.

What DC? Are you sure, that it's not only the last week's PR update?

8:51 am on Oct 6, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member whitey is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 28, 2005
posts:3072
votes: 27


Massive bot activity in last 24hrs [ Google ]
Site:tool - not working properly
Site:tool - results showing mixture of old and new data ie old meta title , current description , correct URL or v.v.
Site:tool - Disparity yesterday between regional results [ looked perfect ] and .COM [ wild and wrong ]
Site:tool - shows screwy "supplementals" returning after they were fixed [ i thought ]
PR toolbar - lowered on one site yesterday
PR others - steady
No change to traffic
Still showing filtered results

[edited by: Whitey at 8:53 am (utc) on Oct. 6, 2006]

9:03 am on Oct 6, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Dec 17, 2004
posts:237
votes: 0


Massive googlebot activity since this morning, although yahoo seems to be crawling heavy as well on all my and my customers' sites...