March Everflux: Google Fresh Listings

Site up for 10 days seems to be added in the recent update

[Adam_C]

From the Deepbot/Freshbot FAQ [webmasterworld.com]:

4. How do you tell the difference between the deepbot and the freshbot.
The deepbot and the freshbot uses different IPs.
The Deepbot uses IPs which run from 216.*
and the Freshbot uses IPs which start with 64.*

Here's something interesting...

A site I put up about 10 days ago seems to have been included in the recent update, but has only been crawled by freshbot - the 64's.

Anyone got any experience of this happening? Or is this a fresh listing without a date? Or something else?

[edited by: Adam_C at 1:07 pm (utc) on Mar. 11, 2003]

[vitaplease]

Fresh listings without date IMO.

At the moment I also have fresh listings without a date, that is, for totally new pages not deepcrawled yet.
(if I remember correctly the last time they carried a Fresh date was march 8th}

[Grumpus]

I'm not at the machine with my notes right now, but it appears that last month Google added a string of 64's that are actually the deepbot.

By memory, it's the 64.68.80.* and 64.68.81.* that seem to be the "Deeps" coming in from the 64 range. Those could be wrong, though.

G.

[ciml]

I agree with vitaplease. Eric Schmidt indicated last year that Google would become fresher and they certainly did.

IMO it's reached the point where only Webmasters would notice that there is an update at all.

[Adam_C]

Grumpus - the same bot hit the site in question on 4 separate occasions:

64.68.82.*

[Grumpus]

That's one of the newer ones. I THINK it's a deep crawler. What you're seeing in the rankings is surely the freshbot additions, though. The reason I emphasise "think" above is because it's still a little new to tell and last month's crawling behavior was a little weird. It wouldn't surprise me if we were looking at a "blend" of the two. Freshbot finds a new page/site and puts it in the fresh index. Then, those "80s" come in and do something that's akin to a "bridge" between the fresh and deep indexes to keep it in there until the real deep crawl comes. (That's a complete guess, though it is one of many possible scenarios). The good news is that you're in there. :)

G.

[rincey]

The same occured with one of my domains. I put it online, it was spidered by 64.68.82.* within hours and appeared in the index within 2 days with 31 pages.

Few days later only 2 pages were left. Now, after the dance, we are back to 31 pages (which is still only a part of the spidered pages) but the domain still shows a graybar so I guess it was not really included in the "deep" index till now...

A few hours ago 216.239.46.* was here for the first time, so I guess all will be fine in the end :)

Ad

[Adam_C]

I've just checked back in the logs of another site that I added some pages to quite late in Jan and they got picked up in the Jan update.

The pages went up on Jan 19th, and 64.68.82.* picked them up a couple of days later.

I think Grumpus is on the right lines with his sugestion of deep 64.68's, or a possible gap being bridged between the deep and fresh bots.

<edit reason> Jan update, not Feb - i.e. end of Jan

[edited by: Adam_C at 4:01 pm (utc) on Mar. 11, 2003]

[ciml]

Adam_C, do the pages that have only been fetched from 64.68.* appear in backlinks? If so, then it looks like Grumpus has made an important discovery.

[Adam_C]

Impossible to say now, as the new site doesn't have any PageRank, and the other site's new pages have now been deep crawled (216.*) and included in the recent update.

[vitaplease]

For totally unlisted sites this may be new,

but in my case with a listed sites with some totally new pages:

Fresh listings without date and no-backlinks.

Has this not been in place for a while though, that is, that Fresh pages can still apear in SERPS without a date tag?

[Grumpus]

I just went back through my logs. Updated pages aren't getting hit by anything with that "82" in the IP range, but when I get a brand spanking new page in there, that "82" shows several days after freshbot has been there. I only went back two weeks, so this isn't conclusive, but it's a start.

G.

P.S. The 64.68.80 and 64.68.81 are still a bit mysterious. I THINK they are deep crawlers, but can't tell for certain as there were several hits in that range very late last month and early this month...

[Grumpus]

Heh - now 82 is hitting me like the deep crawler (on established pages). <shrug> who knows. At least there's something fun to explore while we're waiting for the dance to end. :)

G.

[GoogleGuy]

As long as the listings are fresh, does it really matter which crawler is doing it? :)

[Grumpus]

Aw come on, Googleguy. We finally found a hobby to take up while Google's updating that doesn't give us an ulcer! Let us have our fun. ;)

G.

[oLeon]

GG hit the nail on the head:
what is the matter?

I am wondering how long it may take that we understand, that new listings are better than the same over almost four weeks - not absolute for us, but for the users of Google!
If we all work cleanly with all what we learned here in WebmasterWorld and other ressources, does it matter, which spider is hitting our server - the main point is, that our pages are available. And if we do it correctly, we occur within the top listings. One and the same whether with help from the freshbot or deepbot.

The only point might be, that some pages are always in the index (deep) and some not (fresh).
I assume, that this īd be changed soon, e.g. the pages spidered by the freshbot are more stabile in the serps.

[Grumpus]

Our hypothesis here is that there is an IP range that is actually taking some "fresh" results and stabilizing them somewhat prior to their inclusion in the main index. If that's the case it's good news. If it's not the case, then it's a fun game to play while we watch the dance. :)

G.