Is Freshbot now Deepbot?

The line is getting drawn ever thinner

[trillianjedi]

I've seen several postings about this now in the last few days, although this is my first actual experience of it.

I'm being hit very hard by google's freshbot at the moment, and going deep too. At first glance at what is currently going on with the little guys, I had to check and double check that the IP's were 64.... (they are).

It's behaviour, in terms of hard hitting and depth of crawl (it's going through the entire site) is more like the character of the old deepbot.

In fact, it's identical behaviour to deepbot the last time it crawled this site back in April.

I'm interested in hearing from others who are seeing the same.

TJ

[BigDave]

Both crawl and crawler are now coming from freshbot IPs. You will drive yourself crazy trying to figure out what is going on now based on past experience.

[dvduval]

Because this last update was kind of strange, is that why this page hasn't been updated?
[webmasterworld.com ]
I had bookmarked it and was trying to plot the Dominic date. Is there an agreed upon date for Dominic yet?

[trillianjedi]

I would have thought it's pretty easy to store deepbot's crawl results, get freshbot to crawl those, grabbing the PR and backlinks info along the way and shoving into the index on the fly as it already does.

That way deepbot and freshbot can just be left running and there's no "google-dance" required - the PR iterations can be done on seperate machines on deepbot data, then the whole lot is drawn in by freshbot alongside it's normal rounds.

That way, minty fresh and a more regular cycle. Freshbot also knows better than deepbot which pages have actually changed.

A merging of the algorithms if you like.

TJ

[g1smd]

>> I would have thought it's pretty easy... <<

Three Billion web pages.

[shrirch]

Freshbot indexes normally do not update the cache. Or do they?

[parabola]

Yes, they have for a while. I wouldn't pay much attention to the cache Google shows.

[trillianjedi]

Three Billion web pages.

I know, it's quite amazing but they do manage it!

Having over 200,000 (or whatever it is) PC's in a distributed network does help of course...

I wasn't trying to take anything away from the achievement google have made, but what I perhaps should have said is ".... it would not require much in the way of additional resources to do this...."

TJ

[dididudu]

Can anyone confirm that the fresh bots still only like >=PR4 pages? Cause if this is true, then there is no hope for the new sites to ever get into the index... assuming there is no more deep crawl...

Well, for my site (over 100 pages), only 2 pages get crawled once a few days... googlebot never crawled more than 7 pages / day... And I don't see it that often... what would bring in count for if googlebot "likes" the site or not?

[onionrep]

Hi dididudu

I noticed crawler10 in my logs today crawling pages that I know are not in the index (pages were created yesterday), so do therefore, have no settled PR.

IP address was 64******* which, as has been pointed out is supposedly the freshbot.

To me this means, that either deepbot is out and about masquerading as freshbot, or that deep and fresh are now one and the same..

[WebGuerrilla]

I would agree. I put up a link to a ficticious page in order to see if FB would grab it. It did, which would indicate that this FB crawl from 64.68* is actually the deep crawl.

[parabola]

That's an interesting find. I think GG mentioned that he was "pleased" to see that people are noticing the freshbot acting like deepbot.

My best guess is we are in the middle of the deepcrawl and (almost) no one seems to be taking notice!

[steveb]

Prior to two weeks ago, freshbot hit me everyday this year (usually for 20% to 40% of my site) except two. No freshbot three times this week, and only one day did it take over 10% of my site.

If freshbot is supposed to be taking over for deepbot then it is utterly lame. I had been assuming that Google was crawling less because it realized it didn't have the resources to do what it was trying to do. I'd rather see freshbot disappear and deepbot actually do the thing right than have freshbot mucking everything up.

At this point, freshbot has shown no ability to do what deepbot was able to do circa December/January.

[Dolemite]

I would agree. I put up a link to a ficticious page in order to see if FB would grab it. It did, which would indicate that this FB crawl from 64.68* is actually the deep crawl.

I don't see what this indicates...freshbot has always been about finding new links and new pages. Please elaborate.

[hetzeld]

I've had close to 2500 visits from freshbots (64.68.*) in one hour time last night.
I've never seen a freshbot THAT hungry (and fast)! :)
It definitely looks like a deepcrawl.

Dan

[Adam_C]

This thread might be of interest, where the Fresh/deep convergence was discussed back in March:

[webmasterworld.com...]