mcavill
msg:243359 | 10:16 pm on Nov 11, 2004 (gmt 0) |
I've got a site that's uses a variation of your point 2. The index page has a new additions / updated pages section linking to the new content - new pages, which normally sit 2 levels down the navigation tree, usually show up in google's index in a couple of days (ranking however takes a bit longer :P )
|
jo1ene
msg:243360 | 10:16 pm on Nov 11, 2004 (gmt 0) |
| We are finding that Google takes about 20 days to crawl our site, and that Yahoo takes 30. |
|
I would consider this about right. I wouldn't do anything you mused about. I would be concerned about it backfiring. I find that adding new content frequently is the best way to get the bots a-crawling. I don't think that getting a *complete* crawl any faster would be an advantage anyway.
|
jaffstar
msg:243361 | 9:15 am on Nov 12, 2004 (gmt 0) |
Make sure your site is getting linked from sites with decent PR. You should be crawled more frequently than twenty days.
|
George_hu
msg:243362 | 9:47 am on Nov 12, 2004 (gmt 0) |
20-30 days looks too much. If google like your site structure, will revisit every
2 days, not once.... continously. I have at least 4 site, where returning visitor google and other search engines as well too.
|
Zygoot
msg:243363 | 5:10 pm on Nov 12, 2004 (gmt 0) |
I have a news site and Google (and other search engines) visit my site almost every day. New content on my index page is listed in Google mostly 2 days later but links to the specific new content pages takes a few more days. I think that if you put up a lot of new content every day then their will be a good chance that search engines crawl through your site every 1-2 days.
|
JamieBear
msg:243364 | 6:28 pm on Nov 12, 2004 (gmt 0) |
Thank you all so much for your help. Can I please impose on your just once more to nudge me in the right direction to start correcting this? We are linked by many sites with very high PR. We update our content maybe weekly. The dynamic URLs can be a couple levels deep. Given this add'l info -- would adding a site map off the index page for updated content help? (Thanks mcavill for the idea. Should we manually resubmit the home page to Google when we add content? Force a single daily change to content? (Thanks Zygoot) Anything else you can suggest? Or will it take a lot more than these "tweaks"....like changing the site structure entirely to make the dynamic URLS static-looking :(
|
jo1ene
msg:243365 | 11:52 pm on Nov 12, 2004 (gmt 0) |
I thought the point was that it takes the bots 30 days to crawl the *entire* site. Google comes to my site every day but it takes a while to get to every page. Which is it? Now *I'm* confused.
|
JamieBear
msg:243366 | 8:27 pm on Nov 15, 2004 (gmt 0) |
I'm sorry -- you're right, it really isn't a matter of crawling the entire site. Really, bottomline, we're updating or adding specific pages on a approximately a weekly basis, but could be more often -- and we want to insure that those pages are crawled pronto. Add an updated page site map off the index page? Submit the home page manually? And then cross our fingers and hope for the best? Thanks a lot.
|
zebthepilot
msg:243367 | 6:19 pm on Nov 23, 2004 (gmt 0) |
If you are updating your site once a week or so, and you have a lot of pages; make sure google finds the new content quickly and it doesn't take an entire crawl (or two) to find that new content. As previously suggested have a "what's new" page. if it's something major you might add the links directely to the index; otherwise have a page under the index that links to all the new content pages over the last 2-3 months. google should will probably start checking this page more often (works on all my sites) and crawling the new links from that page. good luck.
|
HyperGeek
msg:243368 | 5:50 pm on Nov 26, 2004 (gmt 0) |
"If-Modified"/"Last-Modified" can help a bit.
[webmasterworld.com...]
|
|
