Welcome to WebmasterWorld Guest from

Forum Moderators: mademetop

Message Too Old, No Replies

What's a normal Google/Yahoo crawl time?

20-30 days... Is there a way to speed up SE crawl time?



10:05 pm on Nov 11, 2004 (gmt 0)

10+ Year Member

We are finding that Google takes about 20 days to crawl our site, and that Yahoo takes 30.

It's a pretty large site, hundreds of pages, dynamic URLs and refreshed frequently.

We let the Google and Yahoo find the pages themselves. Our PageRank is 7, with about 650 inbound links from other sites.

Is this the type of lag time that everyone deals with?

Or is there something we can do to improve it, such as:

1) Manually re-submit a few pivotal pages to Google to "goose" it.

2) Add a full site map from the home page (would be pretty chaotic -- and considerably over the acceptable 100 links, would this be frowned upon by the SE's?)

3) Something else entirely?

Thanks so much in advance for replying to this newbie's question.


10:16 pm on Nov 11, 2004 (gmt 0)

10+ Year Member

I've got a site that's uses a variation of your point 2.

The index page has a new additions / updated pages section linking to the new content - new pages, which normally sit 2 levels down the navigation tree, usually show up in google's index in a couple of days (ranking however takes a bit longer :P )


10:16 pm on Nov 11, 2004 (gmt 0)

10+ Year Member

We are finding that Google takes about 20 days to crawl our site, and that Yahoo takes 30.

I would consider this about right.

I wouldn't do anything you mused about. I would be concerned about it backfiring. I find that adding new content frequently is the best way to get the bots a-crawling.

I don't think that getting a *complete* crawl any faster would be an advantage anyway.


9:15 am on Nov 12, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Make sure your site is getting linked from sites with decent PR. You should be crawled more frequently than twenty days.


9:47 am on Nov 12, 2004 (gmt 0)

10+ Year Member

20-30 days looks too much.

If google like your site structure, will revisit every
2 days, not once.... continously.

I have at least 4 site, where returning visitor google and other search engines as well too.


5:10 pm on Nov 12, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I have a news site and Google (and other search engines) visit my site almost every day. New content on my index page is listed in Google mostly 2 days later but links to the specific new content pages takes a few more days.

I think that if you put up a lot of new content every day then their will be a good chance that search engines crawl through your site every 1-2 days.


6:28 pm on Nov 12, 2004 (gmt 0)

10+ Year Member

Thank you all so much for your help. Can I please impose on your just once more to nudge me in the right direction to start correcting this?

We are linked by many sites with very high PR. We update our content maybe weekly. The dynamic URLs can be a couple levels deep.

Given this add'l info -- would adding a site map off the index page for updated content help? (Thanks mcavill for the idea.

Should we manually resubmit the home page to Google when we add content?

Force a single daily change to content? (Thanks Zygoot)

Anything else you can suggest?

Or will it take a lot more than these "tweaks"....like changing the site structure entirely to make the dynamic URLS static-looking :(


11:52 pm on Nov 12, 2004 (gmt 0)

10+ Year Member

I thought the point was that it takes the bots 30 days to crawl the *entire* site. Google comes to my site every day but it takes a while to get to every page. Which is it? Now *I'm* confused.


8:27 pm on Nov 15, 2004 (gmt 0)

10+ Year Member

I'm sorry -- you're right, it really isn't a matter of crawling the entire site.

Really, bottomline, we're updating or adding specific pages on a approximately a weekly basis, but could be more often -- and we want to insure that those pages are crawled pronto.

Add an updated page site map off the index page?

Submit the home page manually?

And then cross our fingers and hope for the best?

Thanks a lot.


6:19 pm on Nov 23, 2004 (gmt 0)

10+ Year Member

If you are updating your site once a week or so, and you have a lot of pages; make sure google finds the new content quickly and it doesn't take an entire crawl (or two) to find that new content. As previously suggested have a "what's new" page. if it's something major you might add the links directely to the index; otherwise have a page under the index that links to all the new content pages over the last 2-3 months. google should will probably start checking this page more often (works on all my sites) and crawling the new links from that page.

good luck.


5:50 pm on Nov 26, 2004 (gmt 0)

10+ Year Member

"If-Modified"/"Last-Modified" can help a bit.

Featured Threads

Hot Threads This Week

Hot Threads This Month