Feb Crawl Has Started

The 216's have started

[peterdaly]

While discussion has started in some other misc threads, the February crawl is underway. Here is an organized place to discuss it more.

I have one site with googlebot requests from 216.239.46.*

-Pete

[Albaba]

wowwwwww finally she on POWERFULL SPEED

-.-.216.239.46.43.crawl2.googlebot.com - - [07/Feb/2003:09:57:28 +0700] "
-.-.216.239.46.184.crawl7.googlebot.com - - [07/Feb/2003:09:57:37 +0700] .
-.-.216.239.46.88.crawl4.googlebot.com - - [07/Feb/2003:09:57:46 +0700]
-.-.216.239.46.48.crawl2.googlebot.com - - [07/Feb/2003:09:58:10 +0700]
-.-.216.239.46.27.crawl1.googlebot.com - - [07/Feb/2003:09:58:11 +0700]
-.-.216.239.46.140.crawl5.googlebot.com - - [07/Feb/2003:09:58:11 +0700]

[Jesse_Smith]

It's bombing me now!

If it shows up as "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" instead of 'crawl2.googlebot.com', does it make any difference?

[wired4fun]

My pr8 site has been hit pretty hard today, over
40,000 pages crawled by deepbot. Started
yesterday, usually lasts 7-10 days for me.

[xlcus]

If it shows up as "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" instead of 'crawl2.googlebot.com', does it make any difference?

No, that's just a difference in the way different web servers log requests.

crawlXX.googlebot.com is the host name
Googlebot/2.1 (+http://www.googlebot.com/bot.html) is the user agent name.

Some logs won't log the host name, just the ip address. Some logs won't log the user agent name. Some logs will log both.

[Albaba]

Just curious

How long time the deep crawling? is it up to 10 days?

thanks

[coolshop]

I am still noticing this deep crawl to be slower than normal (taking less pages in one day). I am wondering if it is just me, or if others may be seeing the same?

Thanks

Amy

[Pegasus]

wired4fun - 40,000 pages? :P

You must be fast typer. :)

What's your site about?

[wired4fun]

pegasus -
The site is dynamically generated from a DB.
Think of it as 40k different items. Unique content
(descriptions) for each item.

coolshop -
Record number of pages crawled per day for me
on the 6th (tens of thousands), still going
strong (although not as many pages) on the 7th,
still crawling.

[Jesse_Smith]

I'm geting about 175 pages deepcrawled per site. Not sure if that's low, there new sites and this is the first month that I've ever watched the deepcrawl.

[troels nybo nielsen]

Welcome to WebmasterWorld, wired4fun

I see that you have no post count. Evidently an administrator decided that this thread does not count. But I hope that you will join discussions in other threads too.

Pegasus
you will be surprised to see how many members have thousands of pages. Others have less than hundred. The diversity in this community is incredible and one of its strengths.

[corpuscle]

I have a few low PR sites that usually get crawled, but I've still not seen googlebot this month.

I'm wondering if the sites are penalized for some reason. I'm still getting surfer SE traffic from last month's crawl and the feb update, but the fact that the crawler isn't here yet is worrying me.

Anyone else still not seen the bot yet?

[darkroom]

I would like to know when would these new deep crawled pages be up on google......i have had good resutls in the january update...i just dont want my rankings to fall...is there anythign that i can do to protect my rankings from falling...or anything that would make them even better..

.any help will be greately appreciated

[brettb]

corpuscle ~

I am in the same boat. I usually get spidered by now, but it has been very slow so far. I am watching everyday, I will send you a stickymail when we start getting hit. Do me a favor and do the same for me when and if your site gets spidered. I hope we are not on a sinking ship together!

[corpuscle]

brettb - I will do! It's a relief to know it's not just me. I hope we both stickymail each other at the same time (although misery loves company, here's hoping we'll have something more positive in common!) :)

[garry]

Hey guys.........if you all cut your sites back to under a hundred pages each, we'd all get deep crawled in 3 days instead of 10, & the Google update would happen every fortnight instead of monthly :D

[optisoft]

Here is a snippet from one of my personal logs today.
<!--
/ ¦no ref url¦ 216.40.246.25
/ ¦no ref url¦ 216.163.52.231
/ ¦no ref url¦ 216.163.52.231
/ ¦no ref url¦ 216.163.52.231
/ ¦no ref url¦ 216.40.246.25
/ ¦no ref url¦ 216.163.53.246
/ ¦no ref url¦ 216.163.53.246
/ ¦no ref url¦ 66.196.72.85
/ ¦no ref url¦ 216.163.53.246
/ ¦no ref url¦ 216.163.53.246
/ ¦no ref url¦ 66.196.72.23
/ ¦no ref url¦ 66.196.72.22
/ ¦no ref url¦ 216.40.246.25
/ ¦no ref url¦ 66.196.72.81
/ ¦no ref url¦ 66.196.72.86
--!>

These hits were coming each 30 seconds.
PR3 page at the moment- really hope i see higher PR around 5ish.. which is where im grouped in the crawl que it seems.
/pray

[Stefan]

Congrats optisoft. Google and Ink are doing a pas de deux on your site. Good luck with the PR5.

[KakenBetaal]

Mods, would it be possible for us to have a single unified deepcrawl thread like the update thread? I think this thread must be the 6th or 10th thread started on this subject.

It's a mission to find the most appropriate deepcrawl thread to post our comments on, and all the extra threads push the other valuable topics down the page toward oblivion.

[Jesse_Smith]

Any one else still geting attacked by the deepcrawl? It's still geting me. It's got over 1,000 of my vBulletin pages for the first time, thanks to dumping the session ID!

[xlcus]

Any one else still geting attacked by the deepcrawl?

Yep, my server's still getting a strong 20 requests per minute here. I think it's making up for the very slow start! :)

[KakenBetaal]

Still going strong here, although I have the feeling that most of the deep crawl is over because the bot is slowing down again.

Congratulations, Jesse_Smith, good one on removing the sid for Google on your forums! Have any of your forum pages made it into the index via the freshbot yet?

[uber_boy]

If my site's any indication, the deep crawl is still in full swing. That said, googlebot seems to be behaving a wee bit differently this time around. Whereas in the past I'd have a couple of "light" days and then about two days of "terror" as googlebot gobbled up as many pages as my server could give it, this time around the volume has been steadily increasing each day. I was serving up about 500 pages/hour on Saturday and that has steadily increased to the point where today I'm feeding ole googlebot about 1200 pages/hour.

[notbound]

Crawl consistent at over 4000 sessions per hour for over 5 days! WOW! Full crawl? This is also a result of multiple URL submissions to google, that just forward back to my primary domain.

[Yidaki]

>4000 sessions per hour for over 5 days
> a result of multiple URL submissions to google, that just forward back to my primary domain

wow, that's fast. You can expect less crawls in the future, i guess ... ;)

[notbound]

Currently on one of my sites (over 40K pages)
1548 active sessions (all googlebot)
4,651 sessions/hour
77 sessions / minute
Googlebot has been crawling this site since Saturday, 02/02/03. Server's been reset several times, pages changed just can't shake googlebot. This particualr site is also on topic at [webmasterworld.com...]

[Jesse_Smith]

:::Congratulations, Jesse_Smith, good one on removing the sid for Google on your forums! Have any of your forum pages made it into the index via the freshbot yet?

Yep, got's about 60 links there, about every message board index plus a few other pages. They have been listed for atleast four days. Instead of dumping them at the next fresh update, it just keeps updating them. So if your very lucky, it treats you like your listed in the main database! (Might be because the site is listed in ODP.)

Last updated according to Google: Feb 10, 2003, and Feb 11, 2003.

As of about 2:00 AM this morning, it's deepcrawled atleast 1166 pages.

[edited by: Marcia at 1:16 am (utc) on Feb. 14, 2003]
[edit reason] removed link, no specifics please [/edit]

[SubZeroGTS]

so is it normal for google to just send mostly freshbots and a few deepcrawlers my way if it already crawled like 13000 pages last month with deepcrawl?

[KakenBetaal]

I think that if your site had a 13,000 page deepcrawl at the beginning of last month, then you should expect a similar number of pages crawled at the beginning of this month. When did your last deepcrawl happen?

For example, this week I had 7091 pages, last week 6806 pages, and the week before 1518 pages requested by the deepcrawler (216.239.*).

The freshbot has been a lot quieter recently, but normally visits my site most days. This week 544 pages crawled by the freshbot (64.68.*), last week 380 requests, and the week before 1337 requests.