Fresh Tags

What causes you to get/not get them?

[Huntersbiz]

Over the past two weeks almost every result in the top 20 for my two keywords is showing fresh tags. Except my site which is #1 or #2. Can anybody help me understand whay would cause my site to be the only one not showing fresh tags? Googlebot has visited every single day and I do have some changes made. Any insight would be appreciated!

[Visi]

rfg I think it is deeper than just 404 tags. Freshbot seems to be acting as a current database checker, going through a certain number of higher ranked sites to ensure that the database is current, and at the same time trying to verify error pages. The main google crawl is just spidering along evry month, adding to the databse. So if you have been crawled the freshbot will check you out to verify the databse. This is the only logical explanation for what we have seen occuring in the past couple of months. Same thing we are seeing from "slurp".

They are not just crawling and updating but verifying their database. If not we wouldn't be seeing the error pages from "bots" spidering a site. Both these bots (slurp and freshbot) are working from a current listing of some sort.

[Visi]

Well a little more confirmation on this. Today 64.68.82.51 revisited, hitting only the 404 pages. Interesting has been updating the regular pages daily, or almost daily this month, then this. So PR related...yes....seems to be some effect...normally only main page related, but goes deeper sometimes, at least one, sometimes 2 levels...and verifying moved pages that are in googles database. Seems to be multipurpose bot now? Maybe that is why so difficult to determine why or not on visits?

[annej]

I don't think it's a matter of how old a site is. I have an old (96) site that gets fresh bot and new date constantly. I'm wondering if PR is the big factor. Or at least one big factor. Does anyone have a PR6 or above page that doesn't get fresh updates regularly? It would be especially interesting to see if a PR6 page that hasn't been changed for months gets updated by freshie.

Anne

[martinibuster]

Does anyone have a PR6 or above page that doesn't get fresh updates regularly?

Yes. I've rearranged major amounts of content, but no fresh tags for this site.

It would be especially interesting to see if a PR6 page that hasn't been changed for months gets updated by freshie.

On another site, I've made tweaks, and seen the fresh tag within 24-36 hours.

What's the difference between the two? The one that get's freshened up receives more traffic. That's probably not the reason, though. I'm trying to think of more reasons.

[edited by: martinibuster at 5:09 am (utc) on April 1, 2003]

[annej]

martinibuster,

So much for my PR theory then.

Anne

[Powdork]

I have two pr 5 sites that get fresh tags every time they come out regardless of whether there have been any changes to the pages. Both are about three years old. I have another that is regularly visited by freshbot (although in a patchy fashion) but has never had fresh tags. It is six years old but recently went through ownership change and a major change in content. It was never expired, however.

[martinibuster]

Powdork, what are the site size patterns between the two?

I'm curious as to whether site size is a determinant to the fresh effect.

[vitaplease]

IMO, a high Pagerank site/indexpage might not show Fresh tags if it has not received any new links towards it for ages?

[martinibuster]

vitaplease
That's a very good hypothesis. I hadn't thought about it but it happens to agree with the two sites I have in mind.

Does it make sense that a site with increasing/fluctuating links would tend to be one that needs freshening? Is there logic to this?

[vitaplease]

Is there logic to this?

If I were Google I would Fresh crawl:

1. The highest Pageranked pages within a site - regardless of change of content, because these pages most likely will eventually contain a link to a really new internal page. (how else find really fresh content if you do not fresh crawl important pages that could contain new links?).

2. Pages that look like sitemaps and active directories (for the obvious reasons)

3. Pages to which new inbound links occur, because people are voting there is something new/topical going on there (hence - also the interest for real time blogging links).

4. Pages which rank in the top 200 Google SERPS for search queries that suddenly show a "burst" [webmasterworld.com] in usage. (A Zeitgeist like database, only much larger and more immediate)

[Visit Thailand]

Remember as well that if you have a Google nocache tag in the page you won't ever get the Fresh Tag.

[rfgdxm1]

>1. The highest Pageranked pages within a site - regardless of change of content, because these pages most likely will eventually contain a link to a really new internal page. (how else find really fresh content if you do not fresh crawl important pages that could contain new links?).

This is the big one. Remember, the purpose of freshbot is to find new pages. That they update the cache of old ones isn't the main idea. Makes sense that a webmaster will add links to new content from the home page, or other high traffic pages. Particularly if the webmaster considers it important, as Google would want to find the most important new pages.

[Powdork]

martinibuster,
Both are rather small, each about fifteen pages. They both used to have higher pr, one a 6 and the other a 22/40 on the saffir-simpson... er the Google Directory pr-ometer (now it's 16/40). The one with the previous pr of six hasn't had any changes in content or links in about four months. The one with a previous higher pr five has had changes in content and external links.
The site that does not receive fresh tags is changed daily, and has received about thirty shiny new links in the past 60 days, some from pages with fresh tags. It is about 35 pages, all indexed.

[martinibuster]

Pages which rank in the top 200 Google SERPS for search queries that suddenly show a "burst" in usage. (A Zeitgeist like database, only much larger and more immediate)

Hmm... that makes a lot of sense, however, I assume that this relates to people who are searching for your product name, or site name.

[Powdork]

<quote>1. The highest Pageranked pages within a site - regardless of change of content, because these pages most likely will eventually contain a link to a really new internal page. (how else find really fresh content if you do not fresh crawl important pages that could contain new links?).</quote>

I've wondered about this, but wouldn't the new link represent a change in content to Googlebot anyway? Or would that be different with dynamic setup or content? I'm not familiar with that.

[Munster]

Could anyone share the IP add that I should be looking out for to tell me that st google has been down my chimney?

[vitaplease]

Yes, a new link would be a change of content. However, how can Freshbot checker know when the webmaster finally decides to add some really new content on a (new) page through this new link? It has to keep on being polite, revisiting and checking this important site/page, waiting for that new link to new, really Fresh, content to finally be placed.

I would guess the Freshbot does a more or less daily check for the highest Pageranked pages of very high Pageranked sites.

The lower the Pagerank of your site and the longer you take to not add new links, the bigger the chance Freshbot will decide to go look elsewhere more often (unless others keep linking to your important pages recently - remember Google has logged the datestamp of every link since 2000 or so).

[martinibuster]

and the longer you take to not add new links

Interesting... My two sites take unsimilar tracks:

The one prone to freshening has constant additions and changes, whilst the one less prone to freshening has been static for a long time.

Perhaps the next update will trigger freshbot?

[Namaste]

Vita, you have said it...I believe that is what Google IS doing. My experience with my own site shows this.

But I believe point 3 works in conjunction with point 1. I used to have a site that was Main page PR 5 and it used to get fresh botten for all the PR5 & PR4 pages. Now it gets freshbotted for all its PR7 & 6 pages. HOWEVER, there is a 2nd site, which is now PR5 and it's PR4 pages do not get freshbotted. I believe this is becuase it's inside pages have no links coming to it from other websites, whereas in site no. 1 there are links coming to many of it's level 2 pages.

The question I want to ask is how to get level 3 pages freshbotted?

Do any of you have level 3 pages freshbotted?

[brotherhood of LAN]

Vitaplease - 1. The highest Pageranked pages within a site ... because these pages most likely will eventually contain a link to a really new internal page..fresh content if you do not fresh crawl important pages that could contain new links

They even have a paper on the subject
Effective URL Ordering [www-db.stanford.edu]

A related one based on pagerank

Figure 1 shows the average PageRank of all pages down-
loaded on each day of the crawl. The average score for pages
crawled on the 1st day is 7.04, more than three times the av-
erage score of 2.07 for pages crawled on the second day. The
average score tapers from there down to 1.08 after the rst
week, 0.84 after the second week, and 0.59 after the fourth
week.

Breadth First Search Crawling Yields HighQuality Pages [citeseer.nj.nec.com]

I'm sure some of you have read it :-), good to know that they wrote about it too....

[vitaplease]

Thanks for those papers BOL,

I think your .pdf print library must be as overcrowded as mine!

The question I want to ask is how to get level 3 pages freshbotted?

Do any of you have level 3 pages freshbotted?

Namaste,

IMO, Its not the level thats important for fresh-crawling. What I find is that one link from my index page or another highlevel internal Pageranked page is enough to get the new page freshcrawled and indexed.

I believe there are basically two forms of Fresh pages [webmasterworld.com](msg#:8), those that get checked for new links to new pages (the highest level Pageranked site internal pages (points 1/2) and the really new content/page Fresh pages(3).

PS. Do a Google search for "feedback" and notice how many pages have the Freshtag. Not that these "feedback-form" pages get changed/updated a lot, they just get a lot of new links (internal in this case) towards them on a continous basis (and/or probably are one of the highest Pageranked within the site).
Also mentioned here: [webmasterworld.com]msg128

[Namaste]

Vita, I know that I can give a link to any page from index and get it fresh botted. How to get pages freshbotted which are not linked from index page?

my index is a 7, all that link from it are 6, all that link from 6 are 5 and so on. How to get 5 and below freshed?

[vitaplease]

Namaste,

The obvious answer you are not looking for would be to get an external page with fresh listing to link to your internal page.

Maybe someone else has tried this, otherwise worth a trial, give your third level PR5 page a link from your index or second/third best page for only two days and see if it gets fresh indexed until the update.

[Namaste]

give your third level PR5 page a link from your index or second/third best page for only two days and see if it gets fresh indexed until the update

it does, immedaitely, I do this sometimes for important pages.
but am looking for some other way to achieve this.

[vitaplease]

Another paper on how many pages regularily change content:
[research.microsoft.com...]

although not very suprising:

A page's previous change rate is a good predictor of it's future change rate.

from Rubble's posting on Microsoft related papers: [webmasterworld.com...]

[brotherhood of LAN]

Any indication that fresh tags are reserved for certain "keywords", maybe like "news"? i.e.

4. Pages which rank in the top 200 Google SERPS for search queries that suddenly show a "burst" in usage. (A Zeitgeist like database, only much larger and more immediate)

Would google look for the word bursts to find the news or find the news and examine the word bursts? ;)

I've seen (and read others) mention competitive terms and how the SERP's are more adaptive, sounds like something like that could be going on.

Care to elaborate Vitaplease? :)

^{Extra freshness for this high-spam-reporting/competitive industry/high search volume/high dollar areas?}

[vitaplease]

Would google look for the word bursts to find the news or find the news and examine the word bursts? ;)

Chicken and egg again.

I'd say journalists are generally slower in grasping (non-news) popularity trends than searchers.

I'd say searchers are generally slower in grasping trends than bloggers.

Google can select their word bursts for all three now.

high dollar areas of keywords?

They could randomise results to show the site owners the advantage of ranking high for a moment - hence advertising adwords afterwords. IMO, that would lead to even more SEO-type spam for regular results which google does not want.

[annej]

The question I want to ask is how to get level 3 pages freshbotted?

Some of my third level pages get fresh tags and some don't. All the third level pages I've checked so far are PR5 linked from PR5 level two pages. The difference seems to be that the third level pages that have even 1 outside link have fresh tags from yesterday. Those that are all internal links don't.

I don't have time right now to check enough to be really sure but I did check several.

Has anyone else noticed this?

Anne

[Namaste]

have been getting fresh tags on level 3 pages since my last post.
I'm sure if this is because of Google's increased generosity, or because I undertook a content update across my site. My bet is that it's the latter.