Forum Moderators: open
This is not whining, I would really like to know even if I have to bite the bullet and make changes. I do not count on Google alone, but it sure would help : )
My understanding ( which is usually wrong ) was that the frequency of visit did not rely so much on how much was on the site, or the number of backlinks, but on the frequency that you add new pages, or change old ones. It is sort of like - the harder you work, the harder googlebot will work.
This raises an interesting point regarding the opportunity for a new site that changes frequently to get indexed at all.
Given:
* A search engine only has so many resources, and has to decide how to allocate those resources.
and
* The results returned by a search engine have to be relevant; otherwise users will lose confidence.
Therefore:
A new site that changes every 5 minutes may never get indexed because it has not earned the call of the crawler every 5 minutes; and because it changes so often the search engine cannot guarantee relevancy to a users' query.
Conclusion:
If you're a new site that is changing frequently; at least have a relevant and static front page that does not.
As an aside; there seems to be some mis-understanding at to what warrants a frequent visitation / re-index by a search engine spider.
The view that "the more important a search engine considers your page the more frequently their spider will visit" is way too simplistic.
Firstly, regardless of your considered importance, if your content doesn't change there is absolutely no point in re-indexing your page every 5 minutes.
The Privacy Statement page at microsoft.com has a displayed Google PageRank of 9; but I doubt that Googlebot visits any more than once a month, if that. Why not? No need; waste of resources. Doesn't detract from its considered importance though and a justified PageRank of 9.
Away from the lofty heights of PageRank 9 (of which most of us here only dream), a search engine still has to manage resources. Regardless of what a search engine thinks of a page (ala PageRank), there is no point in re-crawling a page that doesn't change very often.
Now given that a search engine's results have to be relevant; the best use of your spiders' resources is to try and visit pages with about the same frequency with which changes are made.
This is easy for a spider to "learn". You visit for the first time, and then come back in, say, a week. If changes haven't been made then you increase the time before your next visit. If changes have been made then you might decrease your time before the next visit.
By repeating this simple algorithm indefinitely (incrementing and decrementing the time between visits) the spider will after a few visits be just about in sync with the frequency at which the page updates.
Most efficient use of resources and almost guaranteed relevant results for your users.
[edited by: dmorison at 2:02 pm (utc) on Aug. 21, 2003]
Perhaps I've traded the good of my site for the good of my users - at least until Google visits again next month.
[added]
your addendum was as informative as the body as your message but it appeared while I was writing; I didn't get it the first time around so I'll add this in one of my own. Thanks.
What do you mean by 'should be fine'? Thanks for the reassurance all the same : )
Yet another dumb question: I read (here, from a senior member) someone saying that they could have freshbot 'on (someone else's) site in under an hour' - what was THAT all about? Sounds enticing but one must be careful...
googlebot is working two shifts:
1) reindexing changed pages: seems to be a continuous process now and - my impression, nothing else - appears to be driven by google's index and not by anything I do. Maybe I am wrong but googlebot does not grab all the pages I change but seems to follow its own route.
2) adding new pages to the index: frequency depends on page-rank, so new sites will have to wait for a while i.e. maybe one month and getting as many incoming links as possible will help.
So there must be another answer.
Actually, I know what it was...how would Google know how often you content changed unless it came and looked as often as you changed it? Seems kinda paradoxical...mabye that's what they have two bots for...but someone recently said that they don't have two anymore or something to that effect. "My brain hurts, my brain hurts, my brains hursts...today."
Jordan
Perhaps if incoming links bring Googlebot to my door more than once in a web crawl, he'll send a message to 'DeepGoogle' to pay me a visit. (Oooo! Careful!)
I think that summed up my thoughts fairly well, but as I said, I could be wrong. Thinking about it, I doubt that that is the only factor. Perhaps it is one of many, including the frequency of backlink additions.
My logs do seem to indicate that the more changes I make recently the more I get visited. but I have also been adding backlinks.
Perhaps googlebot has "flavours of the month"!
