Freshbot visits

[tejas_shah]

Is there a magic formula which would make freshbot visit your site and its webpages everyday? My site's index page does get a visit from freshbot everyday but not web pages. I have made changes to web pages, but not showing up in SERP. any suggestions besides me waiting for the deep crawl?

thanks in advance.

TJ

[takagi]

You won't get a freshbot if the page with the highest PR (usually the index page) has a low PR (e.g. PR2). If it is like PR4 or PR5, than one or several pages will be visited by the freshbot. And with a higher PR even more and with a smaller interval.

Freshbot will especially visit those pages that change frequently (weekly, daily), so it helps the make at least some minor changes to some pages on a regular basis, if your PR is high enough to get visits by the freshbot.

[optisoft]

From my experience if you are not on what I call the freshbots favorites list, then it will take around 3-4 weeks of daily updates to make it the list that gets the actual fresh tag. That is assuming freshbot visit's you every 1-3 days to check for your fresh work, which does seem to have extreme relevence to the sites page rank. There is also the whole link web factor, regarding domains that are on the freshbot favorites list, and link you. As well as 1,000's of other factors that I found by spending large amounts of time searching google for random keywords as research.

[tejas_shah]

I am trying to figure out Freshbot and how it works. This is my conclusion: Freshbot only visits the links on the index page, it will not crawl deeper then that. Would you agree with my conclusion?

Also, would it make differnece if i put the complete website address for <a href> link. For example, putting
<a href = "http://www.site.com/page.html"> vs.
<a href = "page.html">, do you think freshbot differentiates b/w those two?

Thanks.

Tejas

[dotremy]

I got a brand new site, new domain and all.
Freshbot visits my main site every 2 days.
I guided freshbot to my new site, and it indexed 2 pages.
2 day's later it revisited, also the earlier added pages, and got from those two to 4 other ones (on the new site)
This ritual repeated itself for about a week or so and there are now 47 pages of the new site in the index (when i click [more results from ....]

That made me think, isn't it something like this?
10 times freshbot equals deepbot? In a way?

My sites refering to the new site are a PR6 and PR7
(one in my own language and one in English, the content is basicly the same (keyword) but sites are both different, not just a translated version, and have also different visitors and members)

[freejung]

>Freshbot only visits the links on the index page, it will not crawl deeper then that.

Well, that has not been my experience. With my older site, yeah, freshbot usually doesn't get more than two layers deep. But with a new site it will go deeper, sometimes crawling the whole site.

And note that I say "usually," sometimes it picks up a 3rd level page on my older site too. I don't think the freshbot's behavior is as predictable as all that. I think there's a random factor which influences how it crawls.

[hitchhiker]

tejas - it could be due to the page type you're using

- dynamic pages with unknown tags? (not sure about this one)
- dynamic pages with >2 querystrings?this=1&that=2&more=34
- constantly moving urls, like having node pointers on the querystring
- sometimes using ID in a querystring has caused problems (perhaps always infact)
- also the age of your site in the index, my new sites get about 11/50 each time, and then work their way up gradually

[quotations]

>Freshbot only visits the links on the index page,
>it will not crawl deeper then that.
>Would you agree with my conclusion?

No, I have to disagree.

I have

index.html which links to

widgets1.html
widgets2.html
widgets3.html
widgets4.html
widgets5.html

Then on

widgets1.html
widgets2.html
widgets3.html
widgets4.html
widgets5.html

I have links to

aboutgreenwidgets1.html
aboutbluewidgets1.html
aboutredwidgets1.html
aboutyellowwidgets1.html

I don't link to

aboutgreenwidgets1.html
aboutbluewidgets1.html
aboutredwidgets1.html
aboutyellowwidgets1.html

from any index pages yet I do get freshbot crawls on those pages.

[tejas_shah]

So is it safe to say that Freshbot goes down 2-3 levels deep? mannn, this thing is so random.

Tejas

[hitchhiker]

if ur using HTML, (sorry missed that post)

<a href = "/"> would be the code for the homepage

<a href = "http://www.site.com/page.html"> =
<a href = "/page.html">

<a href = "page.html"> would only count if the refering page was in the root dir (in this case). But if it was, then in reality it's the same page, google would most probably resolve it like that.

[nadsab]

I am curious about a couple of things. Optisoft said above "it will take around 3-4 weeks of daily updates to make it the list that gets the actual fresh tag. That is assuming freshbot visit's you every 1-3 days to check for your fresh work..."

So does freshbot do two things? One - it visits sites just about every day to check to see if there is new content, and if there is, it "remembers" that the page has been changed, and TWO, it comes back maybe a day later to freshen a few of those pages it remembered (after the 3-4 weeks of updates)? And once it starts coming every day it comes back twice? Once to check for freshness and a second time to update the cache? Because how would freshbot know how to refresh a page unless it remembers this (after the 3-4 weeks of regular visits Optisoft is talking about above)?

dotremy said that he got his whole new site indexed piece by piece starting right away ( I think?). Does anyone think that google will stop doing the monthly deep crawls and start crawling on a several time a week basis, only crawling smaller pieces of sites each time it visits?

Also does a "Deep Crawl" refresh it's cache for all pages regardless of what pages have been modified/changed or not changed?

Does anyone think that sometimes freshbot randomly becomes deep crawl for a few sites, independent of the major deep crawl?