Question about deepcrawl

When it comes multiple times

[mfishy]

One thing I cannot figure out is this. Deepcrawl hit my pages several times over about 10 days last month. If it comes back over and over again over a period of time, what time does it wind up caching for next update? If it is the first, what is the point of multiple visits to the same page?

[markdidj]

I'm not an expert, but maybe some of those came from links to your site? I don't know how long a deepcrawl takes, but if it's got to index the worlds sites, it's gotta take a while. That's my thought.

[nquinn]

Question:

Deepbot hit my PR4 site this morning, but
it didn't crawl all of my new pages (hit maybe
1/4 of them).

Is it coming back for seconds?

I certainly hope all the hard work I did won't have
to wait until next month!

-Neil

[mrguy]

I would agree that when you see deepbot hit a few pages and then leave, it probably came in from a link.

I will come back and crawl your whole site when it is your turn!

In the mean time, take advantage and update, make changes, etc.