homepage Welcome to WebmasterWorld Guest from 54.197.94.241
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Freshbot visits
tejas_shah




msg:34139
 4:52 pm on Apr 4, 2003 (gmt 0)

Is there a magic formula which would make freshbot visit your site and its webpages everyday? My site's index page does get a visit from freshbot everyday but not web pages. I have made changes to web pages, but not showing up in SERP. any suggestions besides me waiting for the deep crawl?

thanks in advance.

TJ

 

takagi




msg:34140
 6:23 pm on Apr 4, 2003 (gmt 0)

You won't get a freshbot if the page with the highest PR (usually the index page) has a low PR (e.g. PR2). If it is like PR4 or PR5, than one or several pages will be visited by the freshbot. And with a higher PR even more and with a smaller interval.

Freshbot will especially visit those pages that change frequently (weekly, daily), so it helps the make at least some minor changes to some pages on a regular basis, if your PR is high enough to get visits by the freshbot.

optisoft




msg:34141
 8:37 pm on Apr 4, 2003 (gmt 0)

From my experience if you are not on what I call the freshbots favorites list, then it will take around 3-4 weeks of daily updates to make it the list that gets the actual fresh tag. That is assuming freshbot visit's you every 1-3 days to check for your fresh work, which does seem to have extreme relevence to the sites page rank. There is also the whole link web factor, regarding domains that are on the freshbot favorites list, and link you. As well as 1,000's of other factors that I found by spending large amounts of time searching google for random keywords as research.

tejas_shah




msg:34142
 8:51 pm on Apr 4, 2003 (gmt 0)

I am trying to figure out Freshbot and how it works. This is my conclusion: Freshbot only visits the links on the index page, it will not crawl deeper then that. Would you agree with my conclusion?

Also, would it make differnece if i put the complete website address for <a href> link. For example, putting
<a href = "http://www.site.com/page.html"> vs.
<a href = "page.html">, do you think freshbot differentiates b/w those two?

Thanks.

Tejas

dotremy




msg:34143
 9:24 pm on Apr 4, 2003 (gmt 0)

I got a brand new site, new domain and all.
Freshbot visits my main site every 2 days.
I guided freshbot to my new site, and it indexed 2 pages.
2 day's later it revisited, also the earlier added pages, and got from those two to 4 other ones (on the new site)
This ritual repeated itself for about a week or so and there are now 47 pages of the new site in the index (when i click [more results from ....]

That made me think, isn't it something like this?
10 times freshbot equals deepbot? In a way?

My sites refering to the new site are a PR6 and PR7
(one in my own language and one in English, the content is basicly the same (keyword) but sites are both different, not just a translated version, and have also different visitors and members)

freejung




msg:34144
 9:31 pm on Apr 4, 2003 (gmt 0)

>Freshbot only visits the links on the index page, it will not crawl deeper then that.

Well, that has not been my experience. With my older site, yeah, freshbot usually doesn't get more than two layers deep. But with a new site it will go deeper, sometimes crawling the whole site.

And note that I say "usually," sometimes it picks up a 3rd level page on my older site too. I don't think the freshbot's behavior is as predictable as all that. I think there's a random factor which influences how it crawls.

hitchhiker




msg:34145
 9:31 pm on Apr 4, 2003 (gmt 0)

tejas - it could be due to the page type you're using

- dynamic pages with unknown tags? (not sure about this one)
- dynamic pages with >2 querystrings?this=1&that=2&more=34
- constantly moving urls, like having node pointers on the querystring
- sometimes using ID in a querystring has caused problems (perhaps always infact)
- also the age of your site in the index, my new sites get about 11/50 each time, and then work their way up gradually

quotations




msg:34146
 9:33 pm on Apr 4, 2003 (gmt 0)

>Freshbot only visits the links on the index page,
>it will not crawl deeper then that.
>Would you agree with my conclusion?

No, I have to disagree.

I have

index.html which links to

widgets1.html
widgets2.html
widgets3.html
widgets4.html
widgets5.html

Then on

widgets1.html
widgets2.html
widgets3.html
widgets4.html
widgets5.html

I have links to

aboutgreenwidgets1.html
aboutbluewidgets1.html
aboutredwidgets1.html
aboutyellowwidgets1.html

I don't link to

aboutgreenwidgets1.html
aboutbluewidgets1.html
aboutredwidgets1.html
aboutyellowwidgets1.html

from any index pages yet I do get freshbot crawls on those pages.

tejas_shah




msg:34147
 9:35 pm on Apr 4, 2003 (gmt 0)

So is it safe to say that Freshbot goes down 2-3 levels deep? mannn, this thing is so random.

Tejas

hitchhiker




msg:34148
 9:36 pm on Apr 4, 2003 (gmt 0)

if ur using HTML, (sorry missed that post)

<a href = "/"> would be the code for the homepage

<a href = "http://www.site.com/page.html"> =
<a href = "/page.html">

<a href = "page.html"> would only count if the refering page was in the root dir (in this case). But if it was, then in reality it's the same page, google would most probably resolve it like that.

nadsab




msg:34149
 5:00 pm on Apr 6, 2003 (gmt 0)

I am curious about a couple of things. Optisoft said above "it will take around 3-4 weeks of daily updates to make it the list that gets the actual fresh tag. That is assuming freshbot visit's you every 1-3 days to check for your fresh work..."

So does freshbot do two things? One - it visits sites just about every day to check to see if there is new content, and if there is, it "remembers" that the page has been changed, and TWO, it comes back maybe a day later to freshen a few of those pages it remembered (after the 3-4 weeks of updates)? And once it starts coming every day it comes back twice? Once to check for freshness and a second time to update the cache? Because how would freshbot know how to refresh a page unless it remembers this (after the 3-4 weeks of regular visits Optisoft is talking about above)?

dotremy said that he got his whole new site indexed piece by piece starting right away ( I think?). Does anyone think that google will stop doing the monthly deep crawls and start crawling on a several time a week basis, only crawling smaller pieces of sites each time it visits?

Also does a "Deep Crawl" refresh it's cache for all pages regardless of what pages have been modified/changed or not changed?

Does anyone think that sometimes freshbot randomly becomes deep crawl for a few sites, independent of the major deep crawl?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved