Crawled again - twice a month?

[zeus]

Google was spidering before and in the update late july why is it in action now?, maybe they what to do a spidering twice a month now.

zeus

[Brett_Tabke]

They have been doing so for the last two years. 2 monthly crawls is the norm.

[rfgdxm1]

My guess has been the second spidering it to specifically delete what is no longer on the web, and to modify the index for pages that have changed changed from the first indexing, and to find and add new pages and index those. Given that within a month most content on the web stays static, there is a lot less processing that needs to be done after the second spidering. The fact that Google doesn't reindex twice a month suggest that it is computationally infeasible (or, at least economically feasible given the computer power it would take) to do so.

[yankee]

The first spidering is for link analysis/PR computation. The second spidering is to update the cache of the page.

[Nick_W]

...and if you get past pr5 you get spidered a little more regularly for the same thing, cache update etc..

Nick

[przero2]

is the bot still going around ... i have not seen in the last 2 days and my site still not crawled fully!

[fiestagirl]

Haven't seen googlebot since the 7th...except for the odd visit for link checking or updating.

[ndaru]

Today's search results show some sites with the latest cache date: 8 Aug 2002. What criteria do Google use to decide which site's cache to update?

[24bit]

Man, My PR5 site still hasn't been crawled yet for the August update. I've had several bots grab a few pages and then leave. I sure hope it's not going to be a grey September.

[przero2]

24bit, same here for a site with PR5 ... grabbed about 25% of pages and did not re-appear for other pages ... don't know what to tell ... my guess is the bot is going to re-visit closer to the update and include them in the index so it gets the most freshest content as this site with PR5 is a quality site ranked very good now!.

[baldrix]

I hope you're right, as this has happened to me this month, too. I have a PR5 site that was crawled methodically down to the very last of the 400 or so pages last month. This month, with a lot of new content and another 50 pages, the crawl took in 100 or so pages in the first couple of days. Since then, nothing except a revisit to robots plus index page.

Scary!

[przero2]

any one see the bot around still except for the robots and index pages?

[Ready To Roll]

Lowly PR3 site here. Last non-index or robots.txt page was crawled 08/07/02 at noon, PST. However, it went through TWO complete crawls, where 100% of the pages were hit. Achieved recent links from PR7 sites; hopefully will make a difference in my rankings :)
R2R

[shelleycat]

any one see the bot around still except for the robots and index pages?

I've had new pages crawled in the last few days. Not just index pages but new content also.

Info that may be relvant: My main page is only PR4 but many of my links weren't in place yet for the last index (first time in for this site) and I'm hoping for a pr5 next time (fingers crossed). I had a deep crawl that grabbed everything over about three days at the end of July and googlebot has popped back a few times since. I add fresh content most days although my overall site is still small. Maybe being new in the index makes a difference?

[przero2]

googleguy, i am a little surprised that a PR5 site of mine (as well as others mentioned in this thread) were not fully crawled this time around. I was hoping that the bot would come over this weekend to get the latest content from these sites but my hopes were dashed!.

is there still a plan for deep crawling the sites before this month's update or start expecting the worse?. thanks