Msg#: 28708 posted 9:59 am on Mar 23, 2005 (gmt 0)
I maintain a large network of websites in alot of different areas so i have quite good insight in crawling activity between different search engines. I find it strange that googlebot is so passive. A normal day i get about ten times more bot hits from MSN spider and about five times more bot hits from yahoos spider. I also see that MSN is much faster including new sites / pages in then google. Is googlebot crawling "smarter" or are they loosing the freshness of their index..?
Msg#: 28708 posted 12:42 pm on Mar 23, 2005 (gmt 0)
I got some very well linked sites that are getting fetched daily by google but the number of fetched pages compared to especially MSN and also Yahoo are much lower.. so the point is that both msn and yahoo includes more fresh pages faster then google the way i see it..
Msg#: 28708 posted 4:49 pm on Mar 23, 2005 (gmt 0)
I always thought, frequency of Googlebot visits depends on PageRank and number of quality inbound links on other high ranked pages. Higher the PR and inbound links greater is the frquency of visits. I am damn sure the WebmasterWorld has a lot more visits and pages get much faster, because of its high PR and quality inbound links. PR5+ sites do usually get visited everyday or atleast every alternate days.
Msg#: 28708 posted 5:06 pm on Mar 23, 2005 (gmt 0)
Google seems remarkably fast respidering my site, usually catching textual changes within a few days at most. And I don't see a serious bandwidth issue from any spiders though I try to turn off some of the more marginal but CPU-expensive features on pages when I detect that the visitor is a spider rather than a human.
However, Google seems to be lamentably (>6 months?) behind on spidering images, which from what I hear is not untypical.