Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: open
BTW: I asked Jeeves but I was referred to WebmasterWorld - a good system ;) - but this forum didn't helped me further.
Anyone knowing the answers? Please post here and mail Jeeves...
Their crawl/update pattern is not like others. They do a partial update / crawl every few weeks, with a full system crawl every 3-5 months, though occasionally some back to back crawls (full) over a couple of months, though this was when Direct Hit technology was absorbed.
> Anyone knowing the answers? Please post here and mail Jeeves.
Don't worry about AJ, they know they're own crawl schedule, unfortunately they don't tell any of us. So that's why we observe. If you look back through the crawl update posts in this forum, you can find substantial information about the crawls, and the developments at Ask Jeeves and their search system TEOMA.
Try a google (or ask jeeves for that matter!) search for "web file analysis software".
If you can download the raw logs from your ISP, there is an abundance of software (some free GPL if you have linux) that will interpret the logs for you and automatically tell you about which bots have visited, give you graps and stats etc.
NetTracker are currently running a 15 day trial with theirs which is Windows based if you want to have a play with something before you decide what to buy.
I've had over 180,000 pages crawled by Teoma in the last few weeks, and she hasn't stopped yet. At times she bangs away at the rate of 8 to 12 pages *per second*.