homepage Welcome to WebmasterWorld Guest from 54.166.105.24
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
Forum Library, Charter, Moderators: mademetop

General Search Engine Marketing Issues Forum

    
What's a normal Google/Yahoo crawl time?
20-30 days... Is there a way to speed up SE crawl time?
JamieBear




msg:243358
 10:05 pm on Nov 11, 2004 (gmt 0)

We are finding that Google takes about 20 days to crawl our site, and that Yahoo takes 30.

It's a pretty large site, hundreds of pages, dynamic URLs and refreshed frequently.

We let the Google and Yahoo find the pages themselves. Our PageRank is 7, with about 650 inbound links from other sites.

Is this the type of lag time that everyone deals with?

Or is there something we can do to improve it, such as:

1) Manually re-submit a few pivotal pages to Google to "goose" it.

2) Add a full site map from the home page (would be pretty chaotic -- and considerably over the acceptable 100 links, would this be frowned upon by the SE's?)

3) Something else entirely?

Thanks so much in advance for replying to this newbie's question.

 

mcavill




msg:243359
 10:16 pm on Nov 11, 2004 (gmt 0)

I've got a site that's uses a variation of your point 2.

The index page has a new additions / updated pages section linking to the new content - new pages, which normally sit 2 levels down the navigation tree, usually show up in google's index in a couple of days (ranking however takes a bit longer :P )

jo1ene




msg:243360
 10:16 pm on Nov 11, 2004 (gmt 0)

We are finding that Google takes about 20 days to crawl our site, and that Yahoo takes 30.

I would consider this about right.

I wouldn't do anything you mused about. I would be concerned about it backfiring. I find that adding new content frequently is the best way to get the bots a-crawling.

I don't think that getting a *complete* crawl any faster would be an advantage anyway.

jaffstar




msg:243361
 9:15 am on Nov 12, 2004 (gmt 0)

Make sure your site is getting linked from sites with decent PR. You should be crawled more frequently than twenty days.

George_hu




msg:243362
 9:47 am on Nov 12, 2004 (gmt 0)

20-30 days looks too much.

If google like your site structure, will revisit every
2 days, not once.... continously.

I have at least 4 site, where returning visitor google and other search engines as well too.

Zygoot




msg:243363
 5:10 pm on Nov 12, 2004 (gmt 0)

I have a news site and Google (and other search engines) visit my site almost every day. New content on my index page is listed in Google mostly 2 days later but links to the specific new content pages takes a few more days.

I think that if you put up a lot of new content every day then their will be a good chance that search engines crawl through your site every 1-2 days.

JamieBear




msg:243364
 6:28 pm on Nov 12, 2004 (gmt 0)

Thank you all so much for your help. Can I please impose on your just once more to nudge me in the right direction to start correcting this?

We are linked by many sites with very high PR. We update our content maybe weekly. The dynamic URLs can be a couple levels deep.

Given this add'l info -- would adding a site map off the index page for updated content help? (Thanks mcavill for the idea.

Should we manually resubmit the home page to Google when we add content?

Force a single daily change to content? (Thanks Zygoot)

Anything else you can suggest?

Or will it take a lot more than these "tweaks"....like changing the site structure entirely to make the dynamic URLS static-looking :(

jo1ene




msg:243365
 11:52 pm on Nov 12, 2004 (gmt 0)

I thought the point was that it takes the bots 30 days to crawl the *entire* site. Google comes to my site every day but it takes a while to get to every page. Which is it? Now *I'm* confused.

JamieBear




msg:243366
 8:27 pm on Nov 15, 2004 (gmt 0)

I'm sorry -- you're right, it really isn't a matter of crawling the entire site.

Really, bottomline, we're updating or adding specific pages on a approximately a weekly basis, but could be more often -- and we want to insure that those pages are crawled pronto.

Add an updated page site map off the index page?

Submit the home page manually?

And then cross our fingers and hope for the best?

Thanks a lot.

zebthepilot




msg:243367
 6:19 pm on Nov 23, 2004 (gmt 0)

If you are updating your site once a week or so, and you have a lot of pages; make sure google finds the new content quickly and it doesn't take an entire crawl (or two) to find that new content. As previously suggested have a "what's new" page. if it's something major you might add the links directely to the index; otherwise have a page under the index that links to all the new content pages over the last 2-3 months. google should will probably start checking this page more often (works on all my sites) and crawling the new links from that page.

good luck.

HyperGeek




msg:243368
 5:50 pm on Nov 26, 2004 (gmt 0)

"If-Modified"/"Last-Modified" can help a bit.
[webmasterworld.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved