deepcrawl done?

only 2 hits so far today

[farside847]

Anyone else see this? I was getting 5k hits a day and today only 2, is the deepcrawl done?

[entropy]

I'd say yes. I've been looking at all the logs of every domain on my server (about 50) and it looks like PR0s have been spidered in the past couple of days. No hits from deepbot today, only a highly active freshbot.

[webdevsf]

Still going for me...

216.239.46.128

[Napoleon]

Having had a minor catastrophe over the weekend, I'm certainly hoping for another day or two.... anyone else sharing webdevsf's experience of heavy crawl today?

I sense the dying embers of the crawl, but hope I'm wrong.

[sullen]

It's all going a bit funny for me this crawl.

I have a new site which has been half crawled (very few actual visitors yet so I can actually read the logs) and an old site which hasn't been crawled yet.

So I don't think it's over, no. Unless there's a no-crawl penalty or something (wrongly applied in this case, obviously). <<sudden panic setting in>>

[zuko105]

Am getting the same Sullen.

I think its way too early seeing as how the last deep crawl was only 2 weeks ago.

Zuko

[whiterabbit]

have had these IP's crawling today:

64.68.82.5
64.68.82.6
64.68.82.7
64.68.82.18
64.68.82.28
64.68.82.38
64.68.82.39
64.68.82.47
64.68.82.58
64.68.82.66
64.68.82.67

crawler10.googlebot.com
crawler11.googlebot.com
crawler12.googlebot.com

don't quite understand the significance, but, my site has vanished completely from google serps :(

[speda1]

whiterabbit, those ip's belong to the freshbot, not deepbot.

[jrobbio]

speda1. I keep reading about suggestions that the freshbot is being used for cross purposes sometimes, which confused loads of people the last month or so. I wonder if the same is happening. I had my biggest ever 64.68.82 representation today so who knows what's going on.

[speda1]

I think thats just speculation.

[speda1]

I'm seeing lots of 216's here. Its not over yet.

[farside847]

whoo-hoo! just started back up for me! A page every other second...

[worker]

216.239.46.136 is crawling through my PR1 site

I was wondering when/if it would hit this site when everyone was talking about the deep crawl of the last week or so.

It just started about an hour or so ago, and it's hitting pages every couple of minutes or so, and then it takes a breather and starts up again.

Only about 150 pages to the site, so it shouldn't take too long to finish.

I'm watching to see if it only crawls the previously freshbot crawled pages, or if it picks up all of the pages in place right now.

[deejay]

A new site, three weeks old, one incoming PR4 link - 3 deepbots arrived five minutes ago.

Fiiiiiinally!

[NexDog]

Freshbot was in yesterday, have 20 deep crawlers on the site at the moment - 9-10 days and counting - insane crawl this month. :D

[le_gber]

Deepbot came on the 17th, grab the robots.txt and nothing else. No sign of him since. Site is PR0 and here the robots.txt file:

User-agent: *
Disallow: /folder1/
Disallow: /folder2/
Disallow: /folder3/

Is there anything I should be worried about?

Leo

[farside847]

Well, no deepbot for 3 days, since the 26th.

Last month I show the deepbot finished on 3/20 and
the update started on 4/10, 21 days later. So, should
be expect the update on 5/17?

[mfishy]

Yah, typically the update happens 10-15 days after the end of the deepcrawl.

It seems to me like GG might be experimenting a bit with the way they do things though so nothing would surprise me at this point.

Since the content was less fresh this month than in the past few maybe they will roll out the new index in a week or so. We will see.