Welcome to WebmasterWorld Guest from 18.207.136.184

Forum Moderators: open

Message Too Old, No Replies

Is Freshbot now Deepbot?

The line is getting drawn ever thinner

     
4:18 pm on May 22, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7256
votes: 3


I've seen several postings about this now in the last few days, although this is my first actual experience of it.

I'm being hit very hard by google's freshbot at the moment, and going deep too. At first glance at what is currently going on with the little guys, I had to check and double check that the IP's were 64.... (they are).

It's behaviour, in terms of hard hitting and depth of crawl (it's going through the entire site) is more like the character of the old deepbot.

In fact, it's identical behaviour to deepbot the last time it crawled this site back in April.

I'm interested in hearing from others who are seeing the same.

TJ

12:46 am on May 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 4, 2002
posts:1687
votes: 0


TheAutarch, there was a particularly busy 64.68 freshbot visit on many sites in the last couple of days. GG said something to the effect that "he was glad people had noticed". There has been speculation that the deepcrawl will shift from a monthly cycle to an ongoing process. No one knows except those at Google. Either way, we can expect to have our sites deepcrawled over the next while and hope that it will show up in the serps eventually.
1:02 am on May 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 27, 2003
posts:845
votes: 4


Ok,

I went away for awhile and came back and the same thing is being rehashed over and over and over.

As I see it based on what GG said.

The May update is over. Accept it and move on.

With the exceptions of some filters and a little data here and there, what you see is what you get until the next update around the second week of June. At that time all the freshest data will be brought in. From the latest crawl or past two, who knows, or does it really matter. A fresh crawl will do just fine.

This has been done to lead up to what I perceive as their rolling update. After maybe one or two more as we know, the dance as we know it will go away and we will in essence be dancing all month long as the freshies act as deepbots.

GG eluded to as such with his statements above.

So, see you at the dance :)

1:12 am on May 24, 2003 (gmt 0)

New User

10+ Year Member

joined:Apr 25, 2003
posts:21
votes: 0


Stephan, yes I remember him saying that he was glad people noticed. I asked if that meant the deepcrawl was happening now. I guess he just can't say but I'm with you, thinking this will just be an ongoing process.
4:38 am on May 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:776
votes: 0


>>> With the exceptions of some filters and a little data here and there, what you see is what you get until the next update around the second week of June. At that time all the freshest data will be brought in. From the latest crawl or past two, who knows, or does it really matter. A fresh crawl will do just fine.

I think Mrguy has it there. The only other thing I see is that GG liked it that we noticed FB was DeepCrawling. I am hoping that all the FB data goes in like FB data over the next couple weeks with all the other "adjustments".

dave

6:35 pm on May 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:776
votes: 0


FWIW...

A new site, I just put up for a friend... Linked to it from my PR 7 site two days ago. My index page is now fresh in cache... and a new page (my friends) is in the main directory.

Probably the result for FB, obviously, but it does show that new data IS going into the Index!

Happy LONG weekend to all!

dave

6:37 pm on May 24, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:776
votes: 0


OK, VERY weird.

I just went back and looked again- SAME keywords (and they are VERY obscure, and very site-specific- this guy repairs OLD widgets, and there is a very specific term ("widgetmister conservation") I used)...

anyway, bith his site and my fresh site are not showing...

Sorry to use more bandwidth here, just thought it was strange to be in and out of main index....

dave

6:09 pm on May 25, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 3, 2003
posts:58
votes: 0


I'm not sure why you think Google is currently using February data, parabola, but my investigations certainly don't support this hypothesis. Granted, page ranks fluctuate for a time and then reverted to their levels from April. However, the actual pages in the index, at least as far as I can tell, are not the same as they were at any point in the past.
6:37 pm on May 25, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


For my site, Google was using data from late March for a very long time, but nearly two weeks ago, they suddenly swapped (and -sj was first I think) to data they collected on or about 2003-05-08. This then spread over the next week to all datacentres.
7:29 pm on May 25, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Apr 13, 2003
posts:207
votes: 0


It isn't the pages that are cached that are indexed/cached that are old. It is the back link structure that is old. As backlinks and thus PR have a significant effect on ranking then many webmasters are seeing rankings as they were in Feb/March.
7:41 pm on May 25, 2003 (gmt 0)

Senior Member

joined:Nov 20, 2000
posts:1336
votes: 0


Merlin30 has it. The key 'data' that is actually missing is the data that is used to calculate ranking and PR. That is generally quite old in what we are currently seeing.

It sounds like that is going to be brought in for around mid-June.

7:53 pm on May 25, 2003 (gmt 0)

New User

10+ Year Member

joined:May 5, 2003
posts:7
votes: 0


In my case Google definitely reversed the caches to the February update. I know because we did a redesign in April which was picked up by deepbot from the logs.

Previously ranked pages are still listed in the SERPS, but do not have any cached infos in them despite being deepbotted.
Some pages reflect newer changes (freshbot activity I assume), but it still looks like a mess.

7:54 pm on May 25, 2003 (gmt 0)

New User

10+ Year Member

joined:May 22, 2003
posts:5
votes: 0


Right now the cache is showing April's data for me.
9:29 pm on May 25, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 8, 2003
posts:103
votes: 0


Uber boy,

Errrm, no.

The data that is old is the backlink data which is the integral part in ranking sites. Most believe the backlinks showing are from Feb crawl. GoogleGuy has even admitted that they have not used recent deepcrawl data for this update. This is NOT a point of debate. Cache means nothing and new pages may appear as a result of freshbot.

We will see new data come in during the next update.

9:34 pm on May 25, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 8, 2003
posts:103
votes: 0


>>TheAutarch, the last deepbot crawl data can be used for cross-verification. It's also there as a safety-net back-up, but I don't think we'll need to use it. <<

This is what GoogleGuy said when asked if April deepcrawl data will be scrapped.

The backlinks data is old.

10:12 pm on May 25, 2003 (gmt 0)

New User

10+ Year Member

joined:May 15, 2003
posts:8
votes: 0


Deepbot checks robots.txt:
crawl27.googlebot.com - - [25/May/2003:14:27:02] "GET /robots.txt

The bots are allowed in, and so it goes:
crawl27.googlebot.com - - [25/May/2003:14:27:02] "GET /first directory/some file.php

But it seems that Deepbot isn't alone:
crawler11.googlebot.com - - [25/May/2003:14:32:22] "GET /second directory/some other file.php

Freshbot didn't even bother to check robots.txt, it just continues where Deepbot left:
crawler12.googlebot.com - - [25/May/2003:14:53:51] "GET /first directory/some other file.php

Ain't they cute? It's almost as if they were one and the same now.

This 211 message thread spans 15 pages: 211
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members