deepbot schedule

analysis of one of our domains

[plasma]

...
04/Dec/2002:01:14:34
06/Jan/2003:03:09:34
...
07/Jan/2003:13:08:02
07/Feb/2003:09:40:36
...
11/Feb/2003:03:32:49
12/Mar/2003:05:52:26
...
16/Mar/2003:05:12:48
16/Apr/2003:15:22:06
...
20/Apr/2003:14:47:48

So I guess deepbot will reach us (this is a .net domain located in germany) around 20/May/2003

[edit]What I find interesting in particular is that the end day of n-1th crawl is ~ the start day of nth crawl.
Is this true for sites that took longer to crawl, too?[/edit]

[jeremy goodrich]

I haven't mapped any correlation with my own sites (too many log files for me to check by hand, and the automated system is in the planning stages at this point).

Though from reading their early research papers, it seems that PageRank values do have some correlation in determining the priority status of any given domain in the crawl order.

[twilight47]

Are you able to correlate the deepcrawl to the update?

[jeremy goodrich]

I've seen the 'deepbot' during the update. So, from my logs, I can't correlate it (as I said, though - too many sites to effecitively monitor one at a time...)

[WebGuerrilla]

I don't think you can make any correlations at this point. Normally,this time of month would be the peak of the deep crawl, but things have been a bit differnt the last month or so.

I think we need to wait a couple of weeks to see if there is a new deepbot cycle.