AskJeeves

Crawling webpages like crazy.

[PhraSEOlogy]

Has anyone noticed that AskJeeves/Teoma bot has been crawling like crazy. My sites have Jeeves as the top bot crawling pages (ususally google - who is crawling like a snail).

Is this just an anomalie or have they ramped up their bot?

[PatrickDeese]

I would speculate that the their stock tumble from MS's preview search engine release has turned up the heat a bit.

AJ is welcome to crawl my sites as much as they want - but I would prefer human visits from AJ to their bot.

[steveb]

Until I see the bot this year, or until I see the other 98% of my competitors sites in the Jeeves index, I'm tempted to consider these crawling like crazy posts to be the equivalent of Elvis sightings.

[PhraSEOlogy]

Yep - I would agree with the Elvis analogy. But I got visited across multiple sites and a new site got 2276 pages requested in a couple of hours. That is probably more pages that have ever been requested by AJ bot in the past two years on all my sites.

Probably wont bring any traffic though!

[asinah]

AJ is crawling around 400,000 pages a month and we do get some good traffic from AJ.

[steveb]

<faint>

The number of pages in the index for one of my sites just went from one, where it has been for many months, to a few hundred. Less than half, but still about 250 times better.

I don't know if this represents anything for the Internet as a whole, but it is good to see.

[Dayo_UK]

Yay - Jeeves update.

I think the crawl was about a week ago too - very good turnaround.

Jeeves is easiest my second biggest referrer behind Google.

I just wish Yahoo would get its act together.

[Swanson]

There was definately an update, traffic to some of my domains has increased by 10 times usual AJ levels.

New crawling has continued - normally updates occur on the 18th (ish) of every month according to my traffic logs.

[b0rdslide]

How is it possible to find out how many pages for a particular site are in the index?

Thanks,
Kev

[Dayo_UK]

Hi Kev

inurl:www.yourdomain.com

should do the trick.

Dayo

[yowza]

Last month AJ visited my new PR3 site and spidered 700 pages in one visit.

I haven't seen AJ since....

[steveb]

<back to reality>

The few hundred pages jeeves managed to finally find from one of my domains maneged to last all of nine whole days. Now POOF they are all gone again.

I guess Jeeves has all that cash on hand because they don't spend 50 cents to keep their hamster-powered search engine from constantly collapsing.

Checking several other sites with just one pfi page, it looks like they have completely removed all pages except the pfi ones.

[andrew_m]

They definitely crawl like crazy. That is they consume about 40% of my traffic for last month and half at least. Average about 65000 hits a day.

And, wow, I get something like 0.04% of human traffic from them.

Gets on my nerves lately to tell you the truth. If there is someone from AJ/teoma here I'd love to hear them..

[RichD]

site:www.domain.com domain seems to provide better results than inurl:www.domain.com - see the results for inurl:www.amazon.co.uk on ask.co.uk for example.

Ask Jeeves has been crawling a few of my sites heavily for the last few months, so much so I had to upgrade my hosting (1.3million hits so far this month on one site). But Ask Jeeves does now provide 33% of the volume of visits that Google does, so it has been worth it for me.

[steveb]

And now poof, pages are back in.

(I searched for text on a bunch of the these missing pages, not just the site and inurl commands.)

[CodeJockey]

Teoma has been lighting us up for about 30,000 combined hits a day for a while now. While the traffic isn't as nice as Yahoo or MSN, it's about equal to Google. The bot is reasonably polite. Hard to turn it off when it's actually returning something.