Crawl Cycle. Pages crawled now, will be in next update?

[bcc1234]

Hey all,

I'm getting lots of hits from googlebot on several sites.

The pages crawled today should appear after the next update which is the end of August, right ?

Just want to make sure I understand the crawl-update cycle.

Thanks.

[taxpod]

You've got it.

[WebGuerrilla]

Yes, in theory. However, it doesn't alwys work that way. There are occasions where you will get crawled, but the new content doesn't make it into the following update.

[bcc1234]

There are occasions where you will get crawled, but the new content doesn't make it into the following update.

How often does that happen ?

And when it does:
- does it affect the whole site ?
- only all new pages ?
- only some of the pages ?

[Brett_Tabke]

It's hit and miss. I think stuff being crawled right now _won't_ make it into the next update. Things crawled before the update, _will_ make it in. High ranking pages that get spidered from now until the next update _probably_ will make it in, but not the entire site.

[shelleycat]

I was hoping the pages google found deep crawling my site yesterday would make it into the next update.

My site was deep crawled around the 7th of July. I know the pages found in that early-July crawl were added in the last index (I just checked to be sure). I also know there was nothing in there before because this is my first time listed with that site. After that deep crawl googlebot popped in and looked at my index page a couple of times until mid-July, and then nothing at all until the deep crawl yesterday.

So realistically the only new material google has for the next update is from the crawl cycle going on now.

This new domain is my first time with log files and such so I don't have much experience. However it seems to me that surely they would want to put in the pages they just found at the next indexing rather than waiting til the one after? What do the more epxrienced people think? (I love that there are experienced people I can ask :))

The main reason I'm wondering about all this is I've realised I'm going to have to move one of my larger sets of pages to a new domain, new url and all, and am trying to suss out the timing to some extent. Hopefully I can move it without being left out of the loop too long.

[ferrari360]

last month a new site of mine was spidered 7 days after the update and it made it into the last update perfectly. I've generally found this has happened with all my sites - If you are getting spidered now.. or in the next few days my guess is that you will make it into the end of august update.

[bcc1234]

If you are getting spidered now.. or in the next few days my guess is that you will make it into the end of august update

I hope so...

[yankee]

"I think stuff being crawled right now _won't_ make it into the next update."

I strongly disagree.

[wasmith]

I'm also in the boat hoping for an upset (update ... including caching/counting some new links) in about 25-28 days.

[teeceo]

I changed some things on my site today "after the google crawl and up- date". my question is, when googlebot crawls my site agian, will it use the "new" keywords that I put in there (after the update) in the update when its done(the august update) or has my site been updated for augast already and I will have to wait another month before my new page is posted? Thanks all

teeceo.

[nutsandbolts]

Yes, you should be in the next update if you're crawled now. This has been true for two of my sites during the last 2 update cycles. Google will keep on crawling for the next update till about the 4th(ish) and then anything after that will be lightly skimmed into the database (so you might get the 0 rank which means it ISN'T a penalty - just the fact Google has touched your index page but hasn't gone in deep - but will crawl you properly next time)

I am always right, I've never been wrong. ever. honest. no, really. :)

[Abrexa_UK]

Brett and Shelleycat, I am fairly confident that the currently crawled pages will make it in to the index. Googlebot is still doing its business on our sites - it did some serious server abuse yesterday (around 50K page requests), and is still going strong today (about 20K so far).

This isn't the same as we have seen before - I think that Google may have altered the crawl routine slightly - because normally our sites get spidered towards the start of the cycle.

Also Googlebot has been acting strangely for the last two weeks, doing some minor crawling (not just the index page) for some sites, but completely ignoring others for what seemed like ages.