Deepcrawl One Pass or More?

[drogers_wxm]

Hi All,

Once again forgive a newbie question but the Google Deepcrawl are we to expect one pass of the site i.e if its been and gone tough wiat another month.

Or should I be still getting my changes made to try and plan for the next dance?

Dave

[troels nybo nielsen]

Dave, we expect you to pay back by learning, learning and learning and then helping the next generation of newbies. :)

Googlebot has begun the maincrawl for updating the index and you cannot expect to have your websites spidered more than once this month. The results from this maincrawl will show in the next update in about 3-4 weeks or perhaps in the following one in 7-8 weeks.

But then there is the freshbot to hope for. Do a site search on the term if you are not familiar with it. There are lots of threads to read.

Troels

[Nick_W]

<silly nitpick>Didn't we have 2 deep crawls last month, one directly after the other?</silly nitpick>;)

Nick

[troels nybo nielsen]

Are you talking about the strange incidents in the days between Christmas and New Year?

[JLindsay]

It seems that the deep crawler comes several times over a few days at the start of each month and so it is not necessarily the case that if it came once you have missed your chance for another month. But you will have to move VERY fast if you do want to be in with a chance (i.e. stop reading this now and get going :) ) It is also worth noting that I also saw the deep crawler come again around the middle of the month.

(By the way, I hope that this might be the case again. I just missed going live with our new site and saw that the deep crawler had already been twice and so rushed to put it up this evening in the hope that I might get a third visit from the 216. tomorrow! PLEASE! PLEASE!)

[Nick_W]

Are you talking about the strange incidents in the days between Christmas and New Year?

No, but possibly it was longer ago. I distinctly remember there bein confirmations of a double deep crawl a while back...

Nick

[drogers_wxm]

Hi ALL,

Thanks for the comments so far just a quick couple of questions.

I was visited early today by: -

216.239.46.77 Googlebot/2.1

and later by: -

64.68.82.74 Googlebot/2.1

Am I right in thinking the first is the deep crawler and the second is the freshbot looking to update the Google cache for my site?

Secondly from my readings of past posts (see I do dig first) am I correct in thinking that the deep crawl is really intended to discover the structure of the site for later crawl by the freshbot?

If that is the case providing I have the structure sorted before the deep crawl but have changed the content post deep crawl but pre-freshbot (you still with me) I should be OK?

Wow did it!

Dave

[Nick_W]

Looks like you're in to me!

Nick

[aspdesigner]

drogers_wxm -

Yes. No. Maybe. ;)

YES about the type of crawlers.

NO about the purpose of the deep crawler. The deep crawler indexes all of your pages for the next update. It did this before the freshbot even existed, and continues to do so now. It is still important for calculating inbound links and PageRank, and is how non-freshboted sites get re-indexed every month.

MAYBE on the freshbot. The freshbot can spot changes in your content (and sometimes newly added pages), but ONLY if your site is important enough to get freshboted, typically, this means sites with a decent PR. Even for sites that qualify, freshbot still does not re-calc your PageRank, this is still done once a month during the update.