Welcome to WebmasterWorld Guest from 54.205.65.73

Forum Moderators: open

Message Too Old, No Replies

Is Freshbot now Deepbot?

The line is getting drawn ever thinner

     
4:18 pm on May 22, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7249
votes: 0


I've seen several postings about this now in the last few days, although this is my first actual experience of it.

I'm being hit very hard by google's freshbot at the moment, and going deep too. At first glance at what is currently going on with the little guys, I had to check and double check that the IP's were 64.... (they are).

It's behaviour, in terms of hard hitting and depth of crawl (it's going through the entire site) is more like the character of the old deepbot.

In fact, it's identical behaviour to deepbot the last time it crawled this site back in April.

I'm interested in hearing from others who are seeing the same.

TJ

2:17 am on May 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 19, 2002
posts:3454
votes: 0


Both crawl and crawler are now coming from freshbot IPs. You will drive yourself crazy trying to figure out what is going on now based on past experience.
10:21 pm on May 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 28, 2001
posts:1380
votes: 0


Because this last update was kind of strange, is that why this page hasn't been updated?
[webmasterworld.com ]
I had bookmarked it and was trying to plot the Dominic date. Is there an agreed upon date for Dominic yet?
11:05 pm on May 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7249
votes: 0


I would have thought it's pretty easy to store deepbot's crawl results, get freshbot to crawl those, grabbing the PR and backlinks info along the way and shoving into the index on the fly as it already does.

That way deepbot and freshbot can just be left running and there's no "google-dance" required - the PR iterations can be done on seperate machines on deepbot data, then the whole lot is drawn in by freshbot alongside it's normal rounds.

That way, minty fresh and a more regular cycle. Freshbot also knows better than deepbot which pages have actually changed.

A merging of the algorithms if you like.

TJ

11:13 pm on May 26, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


>> I would have thought it's pretty easy... <<

Three Billion web pages.

3:17 am on May 27, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 14, 2002
posts:422
votes: 0


Freshbot indexes normally do not update the cache. Or do they?
3:39 am on May 27, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 8, 2003
posts:103
votes: 0


Yes, they have for a while. I wouldn't pay much attention to the cache Google shows.
9:55 am on May 27, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7249
votes: 0


Three Billion web pages.

I know, it's quite amazing but they do manage it!

Having over 200,000 (or whatever it is) PC's in a distributed network does help of course...

I wasn't trying to take anything away from the achievement google have made, but what I perhaps should have said is ".... it would not require much in the way of additional resources to do this...."

TJ

7:18 pm on May 27, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 28, 2003
posts:48
votes: 0


Can anyone confirm that the fresh bots still only like >=PR4 pages? Cause if this is true, then there is no hope for the new sites to ever get into the index... assuming there is no more deep crawl...

Well, for my site (over 100 pages), only 2 pages get crawled once a few days... googlebot never crawled more than 7 pages / day... And I don't see it that often... what would bring in count for if googlebot "likes" the site or not?

7:26 pm on May 27, 2003 (gmt 0)

Junior Member

joined:Oct 22, 2002
posts:86
votes: 0


Hi dididudu

I noticed crawler10 in my logs today crawling pages that I know are not in the index (pages were created yesterday), so do therefore, have no settled PR.

IP address was 64******* which, as has been pointed out is supposedly the freshbot.

To me this means, that either deepbot is out and about masquerading as freshbot, or that deep and fresh are now one and the same..

10:22 pm on May 27, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 26, 2000
posts:2176
votes: 0



I would agree. I put up a link to a ficticious page in order to see if FB would grab it. It did, which would indicate that this FB crawl from 64.68* is actually the deep crawl.
10:33 pm on May 27, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 8, 2003
posts:103
votes: 0


That's an interesting find. I think GG mentioned that he was "pleased" to see that people are noticing the freshbot acting like deepbot.

My best guess is we are in the middle of the deepcrawl and (almost) no one seems to be taking notice!

11:25 pm on May 27, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


Prior to two weeks ago, freshbot hit me everyday this year (usually for 20% to 40% of my site) except two. No freshbot three times this week, and only one day did it take over 10% of my site.

If freshbot is supposed to be taking over for deepbot then it is utterly lame. I had been assuming that Google was crawling less because it realized it didn't have the resources to do what it was trying to do. I'd rather see freshbot disappear and deepbot actually do the thing right than have freshbot mucking everything up.

At this point, freshbot has shown no ability to do what deepbot was able to do circa December/January.

11:35 pm on May 27, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 20, 2003
posts:390
votes: 0


I would agree. I put up a link to a ficticious page in order to see if FB would grab it. It did, which would indicate that this FB crawl from 64.68* is actually the deep crawl.

I don't see what this indicates...freshbot has always been about finding new links and new pages. Please elaborate.

9:39 am on May 28, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Dec 5, 2002
posts:219
votes: 0


I've had close to 2500 visits from freshbots (64.68.*) in one hour time last night.
I've never seen a freshbot THAT hungry (and fast)! :)
It definitely looks like a deepcrawl.

Dan

10:25 am on May 28, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Aug 31, 2002
posts:284
votes: 0


This thread might be of interest, where the Fresh/deep convergence was discussed back in March:

[webmasterworld.com...]

This 211 message thread spans 15 pages: 211
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members