Google Deepish Crawl is on

For those whose idea of sport is tail -f /path/to/log ¦ grep googlebot

[Clark]

I do believe the Googlebot is doing a deepish crawl right now for those who like to follow it.

[mcavic]

Yep, it's been off and on for me, for a couple few weeks. I don't get hit every day, but some days lots of pages.

I've also had fresh tags yesterday and the day before.

[bobosse]

yeah I think it's up for me to...deep crawling's going on right now.

Bobosse

[Clark]

About 1000 pages so far.

[Seattle_SEM]

I'm skeptical of any "deepish" crawl, and wonder if there will even be another "update" as we know it.

GG?

[Rick_M]

I've gotten a few more pages spidered today (about double what I've gotten most recent days). But about 1% of my pages spidered today have 3 variables, whereas it always used to be no more than 2 variables in my dynamic links. On my site, I don't know that adding a 3 variable will mean more pages or not. Would probably make a big difference for some other sites though.

[the_nerd]

I took out the url-parameters completely - generating a single page for every possible parameter - and ever since google is crawling like crazy - almost the whole site every other day.

[Clark]

This crawl has slowly increased with intensity and has been going on for many many hours now. Although the 2 over the July 4th weekend were more intense and got more pages, this one is a different type of crawl. It is definitely going deeper and going after pages it hasn't looked for since Dominic. I feel the Google love is coming back. Amen.

Thanks GG and everyone @ the plex for your hard work. As much as this process was painful for us I know it's your baby and reading all the venom from some people was probably no fun for you at all. Looks like the worst is over. Welcome back :)

[trillianjedi]

I've noticed it over the last week or two.

Funny thing is, I've noticed some new backlinks coming in too and dropping in and out.

I wonder if they are starting to do calculations on the fly?

TJ

[mipapage]

I'm skeptical of any "deepish" crawl, and wonder if there will even be another "update" as we know it.

Fwiw - I was 'deep crawled' on Saturday the 5th of July, and the pages are now stable in the index - sans fresh dates.

(well, they've been in Google for 2.5 days without leaving and without fresh dates. They did originally appear with fresh-dates (a crawl on the 7th), then just never left after the fresh tags disappeared.)

Sooo... Looks like they were 'updated' without there being an actual update...

[kstprod]

Deepbot all is all over me here, too. This is the 3rd day in a row, and no, I am NOT complaining. :)

The weird thing is that since I don't have a ton of pages, it's like it's cycling through my site getting almost all of them, and then starting over, only to get the same ones again. Like I said, 3rd day in a row like this, with FreshyDeepBot or whatever, coming in between gobbling up pages too.

Another odd thing is that with the pages that Fresh has gotten, some have dates and some don't, although the ones without dates are showing up to date in the cache. Why wouldn't they have a date too?

I can't refer to bot IP's because in my logs, they show as crawlXX and crawlerXX.

[edited by: kstprod at 1:48 pm (utc) on July 11, 2003]

[dnbjason]

I tried posting a Discussion a few days ago to let people know the deep crawl was happening and the Moderators never posted it!

Anyways:
Googlebot been hitting me hard now for the last 5 days, but seems to be slowing. I also noticed it has updated the cache file on alot of other website I been watching.

Any guess on how long tell the next update? :)

[Clark]

I have a theory on the duplicate hits but would like confirmation if someone else has analyzed this.

I noticed the pages that got hit multiple times actually had multiple links to them on the index page of my site. Is it possible that the spider is not consolidating identical links on the designated "freshie" pages at the moment?

[mfishy]

Esmaralda shows backlinks that were added only a few days before the "update".

This leads me to believe that the crawl side of it is more of a continuous process rather than simply a cache from a month old deeepcrawl.

They seem to be constantly collecting data for the next "backlink update".

IMO, we will see plenty of deep crawling over the next few weeks.

[mipapage]

Is it possible that the spider is not consolidating identical links on the designated "freshie" pages at the moment?

Clark,

At first glance, I can say that this theory would stand on my site.

[johnnydequino]

Last google update took place around the 15th. In theory, shouldn't the next one take place this weekend?

jd

[mfishy]

<<Last google update took place around the 15th. In theory, shouldn't the next one take place this weekend?>>

In theory they update backlinks/PR about every month. However, the update seems to have just settled this week. That would be funny if they did since many FINALLY just got their pages back in the SERPS and we are seeing some stability for the first time in a while.

Also, many don't even believe we will have "traditional" updates anymore. Either way, I would guess they will display new backlinks and PR at some point and we can call that the update even if they are factoring in new links and PR on the fly and just not showing us (which I don't think they are).

[tkroll]

I have a quick question that I think is on topic, sorta.

I see Googlebot/2.1 in my logs, but it is only picking up pages it already know about. Is this just "freshbot" revisting known links? Will "deepbot" come later and find the new pages?

Thanks.

[mcavic]

tkroll, yes. I think that Googlebot crawls known pages first, and places the new urls in a queue to crawl later. It might take days or weeks to get to the new ones.

[Clark]

That had been happening to me for weeks but this crawl is getting deeper and deeper and crawling 99% new content for me.

[mipapage]

That had been happening to me for weeks but this crawl is getting deeper and deeper and crawling 99% new content for me.

Same here.

[ulstrup]

I'm skeptical of any "deepish" crawl, and wonder if there will even be another "update" as we know it.

Agree with Seattle_SEM.

Crawling is sometimes intensive, sometimes absent.
Everflux rules, dc switches, algo changes?, etc.
New pages since the e-update are still missing though.

[Clark]

Looks like it was just another 5k (for my site) deep crawl again and is pretty much over. Maybe will come back tomorrow again. Maybe this is the new schedule. Daily minor fresh crawls and weekly 2 day deep crawls. We'll probably need at least another week to see if the pattern holds up.

[incywincy]

googlebot has visited my site 18,172 times pulling 37,607 pages in the last year, i've got a daily history of visits and pages. i like to keep an eye on the bot but i can't see how you can use this information to any advantage.

[bobosse]

Saturday 7/12 night...Googlebot is back! Wahooo...2 times within 3 days and crawling deep >1,500 pages!

Anyone else?

When should the new pages appear in the result pages?

Bobosse

[jeremymgp]

Hi folks,

For me I've been crawled and updated all in one fell swoop, from No.40 to No.20 for my 3 main keywords. Some might say it's everflux, but fresh/deepcrawl and other such terms are so open to interpretation these days they don't carry much meaning anymore. The new rankings have stayed for 4 days, and as far as I'm concerned I've been crawled and updated at the same time.

Best,

Jeremy

[Clark]

Part two of the weekend crawl was a few thousand pages for me. I don't think I was updated though. Time will tell. Would love to hear more people but I think it's too early to draw on the exact pattern.

[xlcus]

My heaviest recent crawl by googlebot was back on the 5th July. Nothing has come close since...

GoogleBot accesses per day
---------------------------
.. 31 01/Jul/2003
. 117 02/Jul/2003
.. 29 03/Jul/2003
26947 05/Jul/2003
. 203 06/Jul/2003
. 847 07/Jul/2003
.3042 08/Jul/2003
... 4 09/Jul/2003
. 277 10/Jul/2003
.1077 11/Jul/2003
... 3 12/Jul/2003

[xlcus]

Thought some people might be insterested in the quick script I used to find the above information... If you're running unix/apache you can use this to count your GoogleBot visits per day...

cat access_log ¦ grep googlebot ¦ cut -d"[" -f 2 ¦ cut -d":" -f 1 ¦ uniq -c

[johnnydequino]

I thought the next dance would occur this weekend - I guess not.

Back about six months ago, could you put a finger on the date? Last month the dance occurred around the 15th.

jd