Welcome to WebmasterWorld Guest from 54.166.191.159

Forum Moderators: open

Message Too Old, No Replies

AskJeeves

Crawling webpages like crazy.

     
3:40 am on Jul 3, 2004 (gmt 0)

10+ Year Member



Has anyone noticed that AskJeeves/Teoma bot has been crawling like crazy. My sites have Jeeves as the top bot crawling pages (ususally google - who is crawling like a snail).

Is this just an anomalie or have they ramped up their bot?

3:47 am on Jul 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I would speculate that the their stock tumble from MS's preview search engine release has turned up the heat a bit.

AJ is welcome to crawl my sites as much as they want - but I would prefer human visits from AJ to their bot.

3:58 am on Jul 3, 2004 (gmt 0)

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Until I see the bot this year, or until I see the other 98% of my competitors sites in the Jeeves index, I'm tempted to consider these crawling like crazy posts to be the equivalent of Elvis sightings.
4:12 am on Jul 3, 2004 (gmt 0)

10+ Year Member



Yep - I would agree with the Elvis analogy. But I got visited across multiple sites and a new site got 2276 pages requested in a couple of hours. That is probably more pages that have ever been requested by AJ bot in the past two years on all my sites.

Probably wont bring any traffic though!

12:03 pm on Jul 3, 2004 (gmt 0)

10+ Year Member



AJ is crawling around 400,000 pages a month and we do get some good traffic from AJ.
8:56 am on Jul 11, 2004 (gmt 0)

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member



<faint>

The number of pages in the index for one of my sites just went from one, where it has been for many months, to a few hundred. Less than half, but still about 250 times better.

I don't know if this represents anything for the Internet as a whole, but it is good to see.

9:21 am on Jul 11, 2004 (gmt 0)

10+ Year Member



Yay - Jeeves update.

I think the crawl was about a week ago too - very good turnaround.

Jeeves is easiest my second biggest referrer behind Google.

I just wish Yahoo would get its act together.

11:34 pm on Jul 12, 2004 (gmt 0)

10+ Year Member



There was definately an update, traffic to some of my domains has increased by 10 times usual AJ levels.

New crawling has continued - normally updates occur on the 18th (ish) of every month according to my traffic logs.

11:50 am on Jul 16, 2004 (gmt 0)

10+ Year Member



How is it possible to find out how many pages for a particular site are in the index?

Thanks,
Kev

9:27 pm on Jul 19, 2004 (gmt 0)

10+ Year Member



Hi Kev

inurl:www.yourdomain.com

should do the trick.

Dayo

10:26 pm on Jul 19, 2004 (gmt 0)

10+ Year Member



Last month AJ visited my new PR3 site and spidered 700 pages in one visit.

I haven't seen AJ since....

9:33 am on Jul 20, 2004 (gmt 0)

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member



<back to reality>

The few hundred pages jeeves managed to finally find from one of my domains maneged to last all of nine whole days. Now POOF they are all gone again.

I guess Jeeves has all that cash on hand because they don't spend 50 cents to keep their hamster-powered search engine from constantly collapsing.

Checking several other sites with just one pfi page, it looks like they have completely removed all pages except the pfi ones.

10:08 am on Jul 20, 2004 (gmt 0)

10+ Year Member



They definitely crawl like crazy. That is they consume about 40% of my traffic for last month and half at least. Average about 65000 hits a day.

And, wow, I get something like 0.04% of human traffic from them.

Gets on my nerves lately to tell you the truth. If there is someone from AJ/teoma here I'd love to hear them..

10:43 am on Jul 20, 2004 (gmt 0)

10+ Year Member



site:www.domain.com domain seems to provide better results than inurl:www.domain.com - see the results for inurl:www.amazon.co.uk on ask.co.uk for example.

Ask Jeeves has been crawling a few of my sites heavily for the last few months, so much so I had to upgrade my hosting (1.3million hits so far this month on one site). But Ask Jeeves does now provide 33% of the volume of visits that Google does, so it has been worth it for me.

11:15 am on Jul 20, 2004 (gmt 0)

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member



And now poof, pages are back in.

(I searched for text on a bunch of the these missing pages, not just the site and inurl commands.)

4:32 pm on Jul 20, 2004 (gmt 0)

10+ Year Member



Teoma has been lighting us up for about 30,000 combined hits a day for a while now. While the traffic isn't as nice as Yahoo or MSN, it's about equal to Google. The bot is reasonably polite. Hard to turn it off when it's actually returning something.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month