Forum Moderators: open
Past month googlebot scanned some files in my site, but last weeks he comes and crawls index and robots.txt and goes away... new pages linked from index (pr 3) are ignored and other listed pages too. It just look at index and robot,
so I wonder if that's the normal behaviour of the bot and next month will crawl my new documents or if something is going wrong...
Help please!
Google bot deep crawl once a month
Not a myth.. more like an extinct animal. Until early this year Google did have a more-or-less clockwork monthly cycle with deepbot. Freshbot (a bot from different IP number range) would also paddle around the surface of the web throughout the month and look for new and fresh content/pages/sites.
Deepbot was reasonably reliable, punctual and predictable. The guy your parents wanted you to date in high school.
Freshy was erratic, shallow, a little unstable, made promises (serp listings) that often didn't last long, and great for a quick thrill. The guy you ended up falling for.
Around April (? I forget exacty when) Google announced a change in strategy - deepbot was retired, freshbot became the all-in-one-superbot. Googlebot behaviour now is a combination of the two, and I think is still going through a bit of a teething stage and seems to be settling slowly into a routine.
On the other hand, unpredictability might be the plan.
Frustrating, yes, but G will get to you in time. Best tactic: keep adding links to increase your crawl priority.
You must be leaving the milk and cookies out;)
It seems to me that DeepFreshBot will scan deep on high PR sites (PR7+) at least once every 3 days (sometime everyday) and pick up practically all changed pages. WW is a good example, it gets new pages/threads added reliably within 3 or 4 days.
I think link structure may also have something to do with this;)
For lower PR sites (4's and 5's) DeepFreshBot seems to add a few hundred pages on each visit, but not do the entire job. It eventually finds them all as long as your PR 5 site doesn't grow faster than Googlebot can give up time to crawl them.
For low PR sites DeepFreshBot is not so hot. The old reliable full crawl once per monthish seems to be history, and therefore those sites with lower PR's can take a long time to get fully crawled.
Oh I wish I was in that position. I get crawled so infrequently that it is a real chore searching to check on visits.
Is it a sympton of Google's current problems that pre April sites get updated almost daily, whereas post April sites, like mine, are lucky if they are updated twice a months?
The sites are too young to recognize the exact deep crawl pattern, but I would expect something like every 3-4 weeks.
