Is Freshbot now Deepbot?

The line is getting drawn ever thinner

[trillianjedi]

I've seen several postings about this now in the last few days, although this is my first actual experience of it.

I'm being hit very hard by google's freshbot at the moment, and going deep too. At first glance at what is currently going on with the little guys, I had to check and double check that the IP's were 64.... (they are).

It's behaviour, in terms of hard hitting and depth of crawl (it's going through the entire site) is more like the character of the old deepbot.

In fact, it's identical behaviour to deepbot the last time it crawled this site back in April.

I'm interested in hearing from others who are seeing the same.

TJ

[Stefan]

TheAutarch, there was a particularly busy 64.68 freshbot visit on many sites in the last couple of days. GG said something to the effect that "he was glad people had noticed". There has been speculation that the deepcrawl will shift from a monthly cycle to an ongoing process. No one knows except those at Google. Either way, we can expect to have our sites deepcrawled over the next while and hope that it will show up in the serps eventually.

[mrguy]

Ok,

I went away for awhile and came back and the same thing is being rehashed over and over and over.

As I see it based on what GG said.

The May update is over. Accept it and move on.

With the exceptions of some filters and a little data here and there, what you see is what you get until the next update around the second week of June. At that time all the freshest data will be brought in. From the latest crawl or past two, who knows, or does it really matter. A fresh crawl will do just fine.

This has been done to lead up to what I perceive as their rolling update. After maybe one or two more as we know, the dance as we know it will go away and we will in essence be dancing all month long as the freshies act as deepbots.

GG eluded to as such with his statements above.

So, see you at the dance :)

[TheAutarch]

Stephan, yes I remember him saying that he was glad people noticed. I asked if that meant the deepcrawl was happening now. I guess he just can't say but I'm with you, thinking this will just be an ongoing process.

[carfac]

>>> With the exceptions of some filters and a little data here and there, what you see is what you get until the next update around the second week of June. At that time all the freshest data will be brought in. From the latest crawl or past two, who knows, or does it really matter. A fresh crawl will do just fine.

I think Mrguy has it there. The only other thing I see is that GG liked it that we noticed FB was DeepCrawling. I am hoping that all the FB data goes in like FB data over the next couple weeks with all the other "adjustments".

dave

[carfac]

FWIW...

A new site, I just put up for a friend... Linked to it from my PR 7 site two days ago. My index page is now fresh in cache... and a new page (my friends) is in the main directory.

Probably the result for FB, obviously, but it does show that new data IS going into the Index!

Happy LONG weekend to all!

dave

[carfac]

OK, VERY weird.

I just went back and looked again- SAME keywords (and they are VERY obscure, and very site-specific- this guy repairs OLD widgets, and there is a very specific term ("widgetmister conservation") I used)...

anyway, bith his site and my fresh site are not showing...

Sorry to use more bandwidth here, just thought it was strange to be in and out of main index....

dave

[uber_boy]

I'm not sure why you think Google is currently using February data, parabola, but my investigations certainly don't support this hypothesis. Granted, page ranks fluctuate for a time and then reverted to their levels from April. However, the actual pages in the index, at least as far as I can tell, are not the same as they were at any point in the past.

[g1smd]

For my site, Google was using data from late March for a very long time, but nearly two weeks ago, they suddenly swapped (and -sj was first I think) to data they collected on or about 2003-05-08. This then spread over the next week to all datacentres.

[merlin30]

It isn't the pages that are cached that are indexed/cached that are old. It is the back link structure that is old. As backlinks and thus PR have a significant effect on ranking then many webmasters are seeing rankings as they were in Feb/March.

[Napoleon]

Merlin30 has it. The key 'data' that is actually missing is the data that is used to calculate ranking and PR. That is generally quite old in what we are currently seeing.

It sounds like that is going to be brought in for around mid-June.

[mauijaws]

In my case Google definitely reversed the caches to the February update. I know because we did a redesign in April which was picked up by deepbot from the logs.

Previously ranked pages are still listed in the SERPS, but do not have any cached infos in them despite being deepbotted.
Some pages reflect newer changes (freshbot activity I assume), but it still looks like a mess.

[version2]

Right now the cache is showing April's data for me.

[parabola]

Uber boy,

Errrm, no.

The data that is old is the backlink data which is the integral part in ranking sites. Most believe the backlinks showing are from Feb crawl. GoogleGuy has even admitted that they have not used recent deepcrawl data for this update. This is NOT a point of debate. Cache means nothing and new pages may appear as a result of freshbot.

We will see new data come in during the next update.

[parabola]

>>TheAutarch, the last deepbot crawl data can be used for cross-verification. It's also there as a safety-net back-up, but I don't think we'll need to use it. <<

This is what GoogleGuy said when asked if April deepcrawl data will be scrapped.

The backlinks data is old.

[WeirdCode]

Deepbot checks robots.txt:
crawl27.googlebot.com - - [25/May/2003:14:27:02] "GET /robots.txt

The bots are allowed in, and so it goes:
crawl27.googlebot.com - - [25/May/2003:14:27:02] "GET /first directory/some file.php

But it seems that Deepbot isn't alone:
crawler11.googlebot.com - - [25/May/2003:14:32:22] "GET /second directory/some other file.php

Freshbot didn't even bother to check robots.txt, it just continues where Deepbot left:
crawler12.googlebot.com - - [25/May/2003:14:53:51] "GET /first directory/some other file.php

Ain't they cute? It's almost as if they were one and the same now.