Next Google Deep Crawl?

What Can We Expect.

[JoeHouse]

Can anyone give me an educated guess as to when we will see the next Google Deep Crawl for June and what can we expect to see?

[captainhannes]

olwen,

64.68.82.54 and 64.68.82.65 resolve to crawler11 and crawler12 -> freshbot
The freshbot always resolves to crawlerXX... - the deepbot always resolves to crawlXX...

Regards,
captain.

[div01]

Nope...just freshie.

[Abigail]

Very brief visit last night - 1/2 dozen pages

[JoeHouse]

I just saw an error server message on www. However when I refreshed the page it was back to normal. Can this be the start of a new update?

[WebGuerrilla]

No, it was just an error.

[SilentPartner]

Perhaps things are indeed getting back to normal:

Upon checking my logs, imagine how suprised I was to see about 1600 page fetches from:

216.239.46.171
216.239.46.226
etc...

Unless I've been working too many hours ( and I have ) and am seeing things ( which I don't think so ), it looks the deep crawler is diving in!

- SP

[ edit ] added the .46 to the IP's [ /edit ]

[edited by: SilentPartner at 3:49 am (utc) on June 13, 2003]

[Dolemite]

Deepbot is 216.239.46.*.

[SilentPartner]

My error, I forgot to include the 46...

- Thanks

[Krapulator]

There has been speculation (which was kind of confirmed by Googleguy in the cryptic way that he does) - that the freshbot is performing the deep crawl this month.

It may already have started (or finished)

[Drew_Black]

One of my sites was visited by the bot-like program from 216.239.45.4 earlier this morning. The requests were strange. Two pages were accessed 8 times each. At 5:24:27 am page 1 was accessed 4 times in sequence from the same IP within a two-second window. At 5:24:30 am page 2 was accessed 4 times within 2 seconds.

Then starting at 6:26:08 am the bot-like creature accessed page 1, then page 2 four times in a row over a span of about 30 seconds. (For a total of 4 page accesses each.)

I think that in my case the bot was attempting to figure out how similar the pages are and if they contain the same content. How have I reached this conclusion? The content of the pages in question are pricing comparisons for "Movie Title" and "Movie Title 2". (i.e. the original and the sequel).

I'm not sure if this helps anyone but I thought it might be significant.

[SilentPartner]

I knew I'd been working too many hours - no deep crawl, I was looking at the wrong data -- wrong time frame.

- SP

[EquityMind]

Just had some nice freshbot rankings today :)

[GoogleGuy]

Just to echo what I mentioned in another thread, I think that a lot of people have noticed that freshbot is crawling deeper into websites where the deepbot used to explore. It's getting deeper into not just fresh content these days. Maybe we should rename it to deepfreshbot (freshdeepbot?).

[jeremymgp]

- I'm going for Fredbot

[GoogleGuy]

I'm sure in a few months we'll be abbreviating it as the dfbot. ;) Then again, who knows? Maybe Fredbot will catch on. I hear that >90% of SEOs believe that Googlebot is a "she" though. :)

[entropy]

GoogleGuy, without saying it specifically, I think you're trying to tell us that Google search will move to more of an "old school style" continuously updated index like Altavista. Googlebot will process your pages when it has time to do so, and new pages will be added as processing allows. Would I be correct in saying that?

I've always felt that this was a better way for a search engine to do things, but maybe it was never considered possible with Google until on-the-fly PageRank calculations were perfected.

[Seattle_SEM]

So is that to say, my friend GoogleGuy, that the "deep crawl" which precedes the update is currently occuring?

I also wonder how this bot now differentiates between content which is immediately published to the live index, and content that is stored for the next update?

Thanks a lot.

[Clark]

GG, of COURSE she's a "SHE". All Powerful. All knowing. fickle, finicky, difficult to please but delightful when achieved. And we have to wait forever for her when she's getting ready.
ME/ducks.

[shaadi]

what are these?

9148 0.05% Googlebot/2.1 (+http://www.googlebot.com/bot.html)
10 0.00% Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)

[Pricey]

10 0.00% Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)

Never seen that before...

[EarWig]

[webmasterworld.com...]
EW

[bakedjake]

we've seen deepbot this week on one of our client sites. about 1200 pages so far.

bizarre behavior really. we usually see deepbot hit most of our sites and client sites within the (relatively) same timeperiod. but i've only seen deepbot hit the one client site.

[mfishy]

Can we get an "official" change of the statement; "more than weeks, less than months" to "more than days, less than weeks"? :)

Seriously, though, knowing that freshbot is now doing deepbot's job put a lot of us at ease. Thanks for the info GG!

[Dayo_UK]

Freshbot does seem to be highly active on my sites at the moment - Fresh Crawl on 10th, these results seem to be in the Serps at the moment - another crawl on 11th - a day of yesterday and another crawl today...

Makes you wonder if all the fresh data does make it into the index as fresh serps - or if it is a deepcrawl in disguise :)

Fredabot :) (Female Fresh/Deep bot - with the added advantage that we can call on Google to Fee the bot!)
:)

[Candleman]

I have been lurking for better than a year now and have learned alot from the sharing of information. I felt I had to join so I could get the GoogleGuy messages as Did others I am sure. I hope to add some theory and insite if possible to this board. There are many knowledgable SEO's here and there are not many items to be covered that have not been covered extensively in these forums.

I think the posting of the GoogleGuy Q&A was great and feel a new indexing will start soon, probably within a couple days. The reasoning behind this is that GoogleGuy seems to have tried to add some closure to the last update, thus making it possible to put to rest some fears about the ranking methods many felt were affecting them whether perminent or temporary.

With closure of the Dominic update, it now seems to me that a new update is soon to follow. It only makes sense to calm the masses before relighting the fire. I do believe the upcoming index will not be the spam ridden index a lot feel it has been experiencing lately, but, rather it will give much more insite to what the final Algo will be. Not the final Algo, but certainly closer the end result we may see in the near future.

I hope the worried have not changed their SEO thinking out of fear that they have been penalized or the practices they have been using have no weight anymore unless they know they do have something to worry about.(Not following true SEO practices)I don't think there should be such a great change as to affect sites that have been on top and deserve to stay with good relevant content, proper backlinks, and offering what the searcher is looking for.

When trying to make a better mouse trap you sometimes dismantle the one you know works, you then figure out through tried and true testing what can I do to make this mouse trap better. You then start to dismantle and manipulate inner parts piece by piece and test each one to see if the performance is better. Once the performance is better and meets your need you begin to rebuild. You rebuild by adding interior pieces first(you don't build from the outside in, you build from the interior working out)Now at this point, you have a solid base to work with and any enhancements can be made to the final product.

The only thing that seems to be stable since May 5th is the amount of Prozac some claim to be taking....LOL

All in all, I think we can come to a conclusion that Google was not broke, rather Google was undergoing an extreme makeover which in the long run will benifit searchers and honest SEO's and keep us on our toes as should be.

Wait it out and see the wonderful changes ahead. :)

[clarksc3]

I am getting the fresh bot 64***

But I have also got a few the 216.** Deep bots lately:

* User Session 216.239.97.227 - IPASS-1BLK (CA,US)
2003-06-03 18:25:20 - user created
2003-06-03 18:25:20 /zinline/inlinewifiareas1.htm

* User Session 216.239.37.5 - Google Inc. (CA,US)
2003-06-04 22:24:43 - user created
2003-06-04 22:24:43 /zinline/inlinehistory.htm

* User Session 216.239.33.5 - Google Inc. (CA,US)
2003-06-07 07:39:54 - user created
2003-06-07 07:39:54 /zthemes/party.htm

Just an FYI

[Dagaz]

Does anyone know which bot 200.82.183.145 is?

[ulstrup]

Welcome to candleman!

As for new unknown IP's / robots, please use the Search Engine Spider Identification [webmasterworld.com] forum.

As for reports of freshbot and deepbot, please note that freshbot runs from 64.68.82.* and deepbot from 216.239.46.*.

Google has several other uses for 216.239.*.* i.e. Froogle, language tools etc.

The way I read Google Guy's postings (thanks GG!) is:

1. an update is comming soon, no time specifics.
2. freshbot is taking over the duties of deepbot.

Further there are indications that this is going to be the last update (dance session) and a more continous update will take over - a logic implication of freshbot taking over deepbot duties.

[Seattle_SEM]

Actually, ulstrup, I got GoogleGuy to give us a hint on when the next update will occur [webmasterworld.com] (start with msg #33).

Essentially, they're trying for June.

[ulstrup]

Seattle_SEM

I did see that, but I don't consider "trying for June" specific, no more than "more than weeks less than months" ;)

So comming soon does it for me.