What's a normal Google/Yahoo crawl time?

20-30 days... Is there a way to speed up SE crawl time?

[JamieBear]

We are finding that Google takes about 20 days to crawl our site, and that Yahoo takes 30.

It's a pretty large site, hundreds of pages, dynamic URLs and refreshed frequently.

We let the Google and Yahoo find the pages themselves. Our PageRank is 7, with about 650 inbound links from other sites.

Is this the type of lag time that everyone deals with?

Or is there something we can do to improve it, such as:

1) Manually re-submit a few pivotal pages to Google to "goose" it.

2) Add a full site map from the home page (would be pretty chaotic -- and considerably over the acceptable 100 links, would this be frowned upon by the SE's?)

3) Something else entirely?

Thanks so much in advance for replying to this newbie's question.

[mcavill]

I've got a site that's uses a variation of your point 2.

The index page has a new additions / updated pages section linking to the new content - new pages, which normally sit 2 levels down the navigation tree, usually show up in google's index in a couple of days (ranking however takes a bit longer :P )

[jo1ene]

We are finding that Google takes about 20 days to crawl our site, and that Yahoo takes 30.

I would consider this about right.

I wouldn't do anything you mused about. I would be concerned about it backfiring. I find that adding new content frequently is the best way to get the bots a-crawling.

I don't think that getting a *complete* crawl any faster would be an advantage anyway.

[jaffstar]

Make sure your site is getting linked from sites with decent PR. You should be crawled more frequently than twenty days.

[George_hu]

20-30 days looks too much.

If google like your site structure, will revisit every
2 days, not once.... continously.

I have at least 4 site, where returning visitor google and other search engines as well too.

[Zygoot]

I have a news site and Google (and other search engines) visit my site almost every day. New content on my index page is listed in Google mostly 2 days later but links to the specific new content pages takes a few more days.

I think that if you put up a lot of new content every day then their will be a good chance that search engines crawl through your site every 1-2 days.

[JamieBear]

Thank you all so much for your help. Can I please impose on your just once more to nudge me in the right direction to start correcting this?

We are linked by many sites with very high PR. We update our content maybe weekly. The dynamic URLs can be a couple levels deep.

Given this add'l info -- would adding a site map off the index page for updated content help? (Thanks mcavill for the idea.

Should we manually resubmit the home page to Google when we add content?

Force a single daily change to content? (Thanks Zygoot)

Anything else you can suggest?

Or will it take a lot more than these "tweaks"....like changing the site structure entirely to make the dynamic URLS static-looking :(

[jo1ene]

I thought the point was that it takes the bots 30 days to crawl the *entire* site. Google comes to my site every day but it takes a while to get to every page. Which is it? Now *I'm* confused.

[JamieBear]

I'm sorry -- you're right, it really isn't a matter of crawling the entire site.

Really, bottomline, we're updating or adding specific pages on a approximately a weekly basis, but could be more often -- and we want to insure that those pages are crawled pronto.

Add an updated page site map off the index page?

Submit the home page manually?

And then cross our fingers and hope for the best?

Thanks a lot.

[zebthepilot]

If you are updating your site once a week or so, and you have a lot of pages; make sure google finds the new content quickly and it doesn't take an entire crawl (or two) to find that new content. As previously suggested have a "what's new" page. if it's something major you might add the links directely to the index; otherwise have a page under the index that links to all the new content pages over the last 2-3 months. google should will probably start checking this page more often (works on all my sites) and crawling the new links from that page.

good luck.

[HyperGeek]

"If-Modified"/"Last-Modified" can help a bit.
[webmasterworld.com...]