Too early for a deep crawl?

[irock]

I was wondering if anyone has got a deep crawl by Google. Is it too early to roll out the red carpet for Googlebot?

[Brett_Tabke]

I think the full crawl was last weekend/week.

[Helpmebe1]

I dont have access to logs..but oh man.. I hope brett is wrong!

[Slade]

I don't have any evidence that it has happened yet, but I'm looking at PR1/2 sites...

[Brett_Tabke]

Just checked again. Had a major crawl a week ago. I thought that was it.

[politicsandlabor]

Brett,

You certainly know more about this than I do, but I've checked the three sites where I get access to "raw" log files and we've just been "kissed on the cheek" -not deeply groped.

So either I'm in BIG TROUBLE or something else is going on.

[bobmark]

The last 2 months I have been crawled between the 2nd. and 11th. of the month with results appearing in the month end update (Oct. 2-7 crawl results are in this update) so if that's your normal pattern I wouldn't be too worried.
In my case I AM worried as I had a change of IP Address due to a server upgrade last month and since then zero daily Googlebot crawls and no deep crawl, so I am really hoping to see Googlebot in the next day or 2.

[Helpmebe1]

bobmark,
Thanks for the info.. I see on cometsystems our cache is from Oct 6th and 7th mostly from what I could see (just a quick look) .. so hoping it hasnt been around an earlier then thursday of last week. Good luck with being picked up this month..hopefull you will be..

[bobmark]

Thanks helpmebe1,
I might as well fold up if I drop out of Google in December which is what will happen if I don't get crawled this cycle.
Sure takes the joy out of the nice surprises I have had in this update.
Hey! and I thought you were taking the weekend off! :)

[Helpmebe1]

Bobmark,
Ohh man... well you have access to your logs or are you like me who has to guess? That sucks ... but I think it has been in the time frame your talkin bout, so hopefully that hasnt changed, especially with the late dance!

Taking the weekend off? Whats you talkin bout Willis? :) Actually I never ever work on a Saturday night but the womans in a mood so I said ok.. Ill work.. so here I am.. getting caught up myself.. finally.. hoping the crawl hasnt come yet either... last couple days I added soooo much new product to the site.. the more recent stuff.. I am in the technology field you could say and was months behind.. so hoping to get the newest stuff in by the end of the month as well... cmooonn down googlebot... munch away on our sites! :)

[Powdork]

I had no visits from 216.239... last week and have only been crawled extensively by the fresh bot on the 28th. I hope it hasn't happened yet.

[Brett_Tabke]

Third check on 5 sites - deep crawled last weekend and week. Not out of the ordinary to have two full crawls a month.

[Helpmebe1]

Thanks Brett.. do all the sites you are referring to have kicking PR? I think that is a factor on how early you get crawled? I know my site is bordering on PR5, well hoping to bump from PR4 to PR5 on the update thats going on now.. could that be why your crawled already, with the high PR that is?

[bobmark]

Thanks for the kind words helpmebe1...gee! I can't imagine why she's in a mood after you offered her to me as an unpaid assistant last night!
sorry Brett...end of personal chit chat on your bandwith :)

oh and yeah helpmebe1 that's from my logs and I know sept and oct crawls were between the 2nd and 11th and my memory is August wrapped up on the 12th. I have no idea if crawls are in order of PR or some number of days from the last or what.

[Helpmebe1]

Bobmark.. I heard once that the higher PR people get crawled first.. thats unconfirmed but something I read somewhere.. Well hopefully the googley eyed bot comes and pays a visit so that Christmas is a good one! Or shall I say New Years too! :)

By the way.. I dunno why she would be offended by me offering to loan her out to people as an assistant.. touchy touchy some people are! haha.. Ok.. sorry brett.. back to business here..

OH PS.. Did you do a favoricon bobmark?

[bobmark]

LOL
yeah, but I have't added the code yet so people are still safe from my cheesy little icon (6 spared from it esterday).
I got so much stuff to do that I put off during the update that I'm swamped.
Can you run cgi scripts on your host? If so I can recommend one that will give you instant logging and it is an auto install.

[edited by: bobmark at 3:40 am (utc) on Nov. 3, 2002]

[Powdork]

Brett,
How do you determine if it is a deep crawl? I generally look for 216.239... and see if its grabbing more than just robots.txt or index.
For me there is no difference as far as the depth of the crawl since everything is in the root. Whenever the fresh bot comes by she usually grabs everything. I checked my logs back to the 25th of oct. Did you have any deep crawls before that?

[Helpmebe1]

Bobmark,
Thanks but I really have no clue on the cgi thing.. I am a rtml and html kinda guy.. defintely a cool little toy to play around with though.. gonna have to make one of those.. trying to stick to the updating thing although I did some nice adobe work last night instead..haha.. god such a slacker when it comes to doing updating.. but hey if people find the site, theyll think its pretty! :) Ok.. off to get dinner and then updated updates updates... Hey.. did ya hear ..update! haha.. otherwise well both be wondering what happened? Why isnt our stuff in here? haha :)

[vitaplease]

I would guess the deep crawl to start wednesday, maybe tuesday for real high PR sites.

I'd be willing to bet my mouse pad on that ;)

[Powdork]

I'd be willing to bet my mouse pad on that

I'll take monday, except that I don't understand why Brett feels differently. He should know but noone else will confirm his findings. Perhaps its just some sort of April Fools joke GG is playing on Brett (after all GoogleGuy doesn't know when April Fools is), sending in 64.68 dressed as 216.239. ;)
Anyway, I'll take monday at 3:16 am, my time, on mysite, of my choosing, confirmed by me; for the mousepad. It can't be just a Google mousepad, but THE mousepad. And I still want that quilt.

[rfgdxm1]

If the deep crawler is that 216.* IP one, I haven't seen her in the logs of either of my PR5 sites. Although the Freshbot for some reason is coming by either daily or at most every other day. Google's Freshbot definitely is a nosy one. Although, I much rather see her keep coming around than avoiding me. ;)

[Go60Guy]

I'm seeing a lot of 64.68 today.

[taxpod]

I've been seeing 64.68 since the dance began. This bot pulls down a couple thousand pages a day by I haven't seen any other bot since even before the dance.

One thing that I think would come in handy is a list of the different G-bots and their recorded IPs. Maybe that is too big a task but being able to match up the IPs with the type of beast would have come in handy right now!

[aaronc]

I've gotten about 83 hits from 216.239 and about 99 hits from the 64.68 bot today.
I'm guessing the deep crawl has started.
It seems weird that the deep crawl bot would harvest less pages than the freshbot.
Anyone have any ideas why that might be?
This assumes the 216 = deep and 64 = fresh.

I've learned tons in the past week. I'm glad I found this place before this recent deep crawl. I've optimized the site I've been working on and it's helped alot in keyword matches and results showing up as far as results showing up in fresh searches.

[Powdork]

Taxpod,
Cut & paste this and check out keymasters post.
[webmasterworld.com...]
aaronc
Welcome to Webmaster World.
I hope your right about the deep crawling.
I know your right about this being an invaluable resource.:)

[edited by: Powdork at 7:00 am (utc) on Nov. 4, 2002]

[Paully]

Brett,
You certainly know more about this than I do...

So either I'm in BIG TROUBLE or something else is going on

Ditto here, with a PR6.

This bot pulls down a couple thousand pages a day

Wow.. I wish I had a couple thousand pages crawled pre day... Whats your secret?

[Powdork]

taxpod, I fixed the above post, sorry hadn't had my meds yet;)

[rfgdxm1]

216.* just started hitting my main site. Along with that good old girl freshbot that for some reason just won't leave my sites alone ever. Since others are seeing 216.* too again, looks like the deep crawl has begun.

[europeforvisitors]

It seems to be starting. Googlebot hit 421 of my pages on Sunday, which is about 300 more than it usually does on a good day when it isn't deep crawling.

[djgreg]

Only for understanding:
The deepcrawl usually starts after the update and crawls the sites, finds the links ... which are used for the next update? Right?