Deep Crawl.

[markus007]

Ask is crawling my site deeper then google has ever done, its fetched over 60,000 pages in the last 24 hours... How often do they come up with a new index?

[Dayo_UK]

Ahh - I thought it had finnised crawling and was looking out for an update, must still be doing its rounds.

In my experience Jeeves has been updating on about a 6 weekly(ish) schedule - although it is more than 6 weeks since the last update, perhaps they want to get a bit more data.

Updates tend to be at weekends too (last 2 or 3 I think) - if it is still crawling it might be to early for this weekend.

[markus007]

ya i was surprised, its fetched over 250,000 pages this week which is more then all other SE's combined have done for the month

[XMLMania]

Wow yeah, Teoma is on a rampage on one of my sites:

88151 Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
71159 msnbot/0.11 (NULLED)
28485 msnbot/0.3 (NULLED)
23009 Googlebot/2.1 (NULLED)

Crazy, this is the first time i've seen it spider so vigorously.

[Dayo_UK]

Jeeves/Teoma has updated today - but for my sites there is a lot more crawled data not listed.

Perhaps more updating is still being factored in......

[markus007]

ya still going strong here, fetched another 60,000 pages in the last 24 hours

[Larryhat]

Seeing this thread, I checked my latest access_log file. Sure enough, ask/Teoma has crawled pretty much my one entire smallish site, (about 130 pages). All this in the span of a single 24 hour period. Crawler hits were nicely spaced, maybe 30 seconds apart, and it pulled in robots.txt repeatedly. It looks well behaved from my little corner of the net.

I'm not sure if there is any cause-effect relation here, but my single 3-letter keyword brought my site up on the first page, #10 out of 1.7 million. I can't complain.

- Larry

[rfgdxm1]

>Jeeves/Teoma has updated today - but for my sites there is a lot more crawled data not listed.

Yep. And if ask.com was the dominant search engine I could get filthy rich doing professional SEO. Ask.com isn't an easy nut to crack, but once you know exactly how to do so it's easy to rank well.

[Dayo_UK]

Has anyone got much data into jeeves following the recent crawl.

The about of data added to the index compared to the amount of pages crawled for my sites was only a very small percentage.

Bit of a waste of bandwidth unless these pages do get added. :(

[Mentat]

Ask Jeews is storming my forum LOFI version.
8000 hiths in the last 24 hours

I've never had links from teoma/ask jeews and this is strange!

[robho]

The AJ/Teoma crawler is also hitting one of my sites quite hard, a page every 2 or 3 seconds, 24/7 (there are a few hundred thousand pages on the site).

Every now and then it triggers my mod_throttle rules (too many pages accessed by one IP in a given time frame). I then return a 503 error code. But it just carries on pounding away, ignoring that for hundreds more pages...

Of course, none of the tens of thousands of pages it grabs per day are indexed yet. It just has one page from the site in the SERPS, a holding page from many months ago.

A key to the search engine wars is comprehesive, fresh results. They seem to be trying a bit hard for the first part of that, and failing totally on the second part.

[darqSHADOW]

AskJeeves is behaving rather strange for my site, eating TONS of bandwidth. (More-so than GoogleBot for less pages.) Here are the stats for both:

Googlebot 71606 1.95 GB
AskJeeves 51414 9.25 GB

Makes little to no sense to me.

DS

[XMLMania]

Maybe the majority of the spidered pages by Googlebot was by the Mozilla 5.0 variant?

This bot supports HTTP/1.1 meaning it supports Gzip pages, sometimes upto 1/8th the transfer size of uncompressed pages.

[darqSHADOW]

That's very possible. I will take a look at the raw logs and see if this is the case. It just seems strange to me that AskJeeves (who never spidered more than a few hundred pages a month prior) is all of the sudden raping my entire site. =)

DS