Welcome to WebmasterWorld Guest from 54.144.79.200

Forum Moderators: open

Message Too Old, No Replies

Is Freshbot now Deepbot?

The line is getting drawn ever thinner

     

trillianjedi

4:18 pm on May 22, 2003 (gmt 0)

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I've seen several postings about this now in the last few days, although this is my first actual experience of it.

I'm being hit very hard by google's freshbot at the moment, and going deep too. At first glance at what is currently going on with the little guys, I had to check and double check that the IP's were 64.... (they are).

It's behaviour, in terms of hard hitting and depth of crawl (it's going through the entire site) is more like the character of the old deepbot.

In fact, it's identical behaviour to deepbot the last time it crawled this site back in April.

I'm interested in hearing from others who are seeing the same.

TJ

BigDave

2:17 am on May 26, 2003 (gmt 0)

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Both crawl and crawler are now coming from freshbot IPs. You will drive yourself crazy trying to figure out what is going on now based on past experience.

dvduval

10:21 pm on May 26, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Because this last update was kind of strange, is that why this page hasn't been updated?
[webmasterworld.com ]
I had bookmarked it and was trying to plot the Dominic date. Is there an agreed upon date for Dominic yet?

trillianjedi

11:05 pm on May 26, 2003 (gmt 0)

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I would have thought it's pretty easy to store deepbot's crawl results, get freshbot to crawl those, grabbing the PR and backlinks info along the way and shoving into the index on the fly as it already does.

That way deepbot and freshbot can just be left running and there's no "google-dance" required - the PR iterations can be done on seperate machines on deepbot data, then the whole lot is drawn in by freshbot alongside it's normal rounds.

That way, minty fresh and a more regular cycle. Freshbot also knows better than deepbot which pages have actually changed.

A merging of the algorithms if you like.

TJ

g1smd

11:13 pm on May 26, 2003 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



>> I would have thought it's pretty easy... <<

Three Billion web pages.

shrirch

3:17 am on May 27, 2003 (gmt 0)

10+ Year Member



Freshbot indexes normally do not update the cache. Or do they?

parabola

3:39 am on May 27, 2003 (gmt 0)

10+ Year Member



Yes, they have for a while. I wouldn't pay much attention to the cache Google shows.

trillianjedi

9:55 am on May 27, 2003 (gmt 0)

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Three Billion web pages.

I know, it's quite amazing but they do manage it!

Having over 200,000 (or whatever it is) PC's in a distributed network does help of course...

I wasn't trying to take anything away from the achievement google have made, but what I perhaps should have said is ".... it would not require much in the way of additional resources to do this...."

TJ

dididudu

7:18 pm on May 27, 2003 (gmt 0)

10+ Year Member



Can anyone confirm that the fresh bots still only like >=PR4 pages? Cause if this is true, then there is no hope for the new sites to ever get into the index... assuming there is no more deep crawl...

Well, for my site (over 100 pages), only 2 pages get crawled once a few days... googlebot never crawled more than 7 pages / day... And I don't see it that often... what would bring in count for if googlebot "likes" the site or not?

onionrep

7:26 pm on May 27, 2003 (gmt 0)



Hi dididudu

I noticed crawler10 in my logs today crawling pages that I know are not in the index (pages were created yesterday), so do therefore, have no settled PR.

IP address was 64******* which, as has been pointed out is supposedly the freshbot.

To me this means, that either deepbot is out and about masquerading as freshbot, or that deep and fresh are now one and the same..

WebGuerrilla

10:22 pm on May 27, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member




I would agree. I put up a link to a ficticious page in order to see if FB would grab it. It did, which would indicate that this FB crawl from 64.68* is actually the deep crawl.

parabola

10:33 pm on May 27, 2003 (gmt 0)

10+ Year Member



That's an interesting find. I think GG mentioned that he was "pleased" to see that people are noticing the freshbot acting like deepbot.

My best guess is we are in the middle of the deepcrawl and (almost) no one seems to be taking notice!

steveb

11:25 pm on May 27, 2003 (gmt 0)

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Prior to two weeks ago, freshbot hit me everyday this year (usually for 20% to 40% of my site) except two. No freshbot three times this week, and only one day did it take over 10% of my site.

If freshbot is supposed to be taking over for deepbot then it is utterly lame. I had been assuming that Google was crawling less because it realized it didn't have the resources to do what it was trying to do. I'd rather see freshbot disappear and deepbot actually do the thing right than have freshbot mucking everything up.

At this point, freshbot has shown no ability to do what deepbot was able to do circa December/January.

Dolemite

11:35 pm on May 27, 2003 (gmt 0)

10+ Year Member



I would agree. I put up a link to a ficticious page in order to see if FB would grab it. It did, which would indicate that this FB crawl from 64.68* is actually the deep crawl.

I don't see what this indicates...freshbot has always been about finding new links and new pages. Please elaborate.

hetzeld

9:39 am on May 28, 2003 (gmt 0)

10+ Year Member



I've had close to 2500 visits from freshbots (64.68.*) in one hour time last night.
I've never seen a freshbot THAT hungry (and fast)! :)
It definitely looks like a deepcrawl.

Dan

Adam_C

10:25 am on May 28, 2003 (gmt 0)

10+ Year Member



This thread might be of interest, where the Fresh/deep convergence was discussed back in March:

[webmasterworld.com...]

This 211 message thread spans 15 pages: 211