Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: open
thanks in advance.
Freshbot will especially visit those pages that change frequently (weekly, daily), so it helps the make at least some minor changes to some pages on a regular basis, if your PR is high enough to get visits by the freshbot.
Also, would it make differnece if i put the complete website address for <a href> link. For example, putting
<a href = "http://www.site.com/page.html"> vs.
<a href = "page.html">, do you think freshbot differentiates b/w those two?
That made me think, isn't it something like this?
10 times freshbot equals deepbot? In a way?
My sites refering to the new site are a PR6 and PR7
(one in my own language and one in English, the content is basicly the same (keyword) but sites are both different, not just a translated version, and have also different visitors and members)
Well, that has not been my experience. With my older site, yeah, freshbot usually doesn't get more than two layers deep. But with a new site it will go deeper, sometimes crawling the whole site.
And note that I say "usually," sometimes it picks up a 3rd level page on my older site too. I don't think the freshbot's behavior is as predictable as all that. I think there's a random factor which influences how it crawls.
- dynamic pages with unknown tags? (not sure about this one)
- dynamic pages with >2 querystrings?this=1&that=2&more=34
- constantly moving urls, like having node pointers on the querystring
- sometimes using ID in a querystring has caused problems (perhaps always infact)
- also the age of your site in the index, my new sites get about 11/50 each time, and then work their way up gradually
No, I have to disagree.
index.html which links to
I have links to
I don't link to
from any index pages yet I do get freshbot crawls on those pages.
<a href = "/"> would be the code for the homepage
<a href = "http://www.site.com/page.html"> =
<a href = "/page.html">
<a href = "page.html"> would only count if the refering page was in the root dir (in this case). But if it was, then in reality it's the same page, google would most probably resolve it like that.
So does freshbot do two things? One - it visits sites just about every day to check to see if there is new content, and if there is, it "remembers" that the page has been changed, and TWO, it comes back maybe a day later to freshen a few of those pages it remembered (after the 3-4 weeks of updates)? And once it starts coming every day it comes back twice? Once to check for freshness and a second time to update the cache? Because how would freshbot know how to refresh a page unless it remembers this (after the 3-4 weeks of regular visits Optisoft is talking about above)?
dotremy said that he got his whole new site indexed piece by piece starting right away ( I think?). Does anyone think that google will stop doing the monthly deep crawls and start crawling on a several time a week basis, only crawling smaller pieces of sites each time it visits?
Also does a "Deep Crawl" refresh it's cache for all pages regardless of what pages have been modified/changed or not changed?
Does anyone think that sometimes freshbot randomly becomes deep crawl for a few sites, independent of the major deep crawl?