Welcome to WebmasterWorld Guest from 54.145.11.9

Forum Moderators: open

Message Too Old, No Replies

AskJeeves

Crawling webpages like crazy.

     
3:40 am on Jul 3, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 20, 2003
posts:408
votes: 0


Has anyone noticed that AskJeeves/Teoma bot has been crawling like crazy. My sites have Jeeves as the top bot crawling pages (ususally google - who is crawling like a snail).

Is this just an anomalie or have they ramped up their bot?

3:47 am on July 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 6, 2003
posts:2523
votes: 0


I would speculate that the their stock tumble from MS's preview search engine release has turned up the heat a bit.

AJ is welcome to crawl my sites as much as they want - but I would prefer human visits from AJ to their bot.

3:58 am on July 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


Until I see the bot this year, or until I see the other 98% of my competitors sites in the Jeeves index, I'm tempted to consider these crawling like crazy posts to be the equivalent of Elvis sightings.
4:12 am on July 3, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 20, 2003
posts:408
votes: 0


Yep - I would agree with the Elvis analogy. But I got visited across multiple sites and a new site got 2276 pages requested in a couple of hours. That is probably more pages that have ever been requested by AJ bot in the past two years on all my sites.

Probably wont bring any traffic though!

12:03 pm on July 3, 2004 (gmt 0)

Full Member

10+ Year Member

joined:July 11, 2003
posts:291
votes: 0


AJ is crawling around 400,000 pages a month and we do get some good traffic from AJ.
8:56 am on July 11, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


<faint>

The number of pages in the index for one of my sites just went from one, where it has been for many months, to a few hundred. Less than half, but still about 250 times better.

I don't know if this represents anything for the Internet as a whole, but it is good to see.

Dayo_UK

9:21 am on July 11, 2004 (gmt 0)

Inactive Member
Account Expired

 
 


Yay - Jeeves update.

I think the crawl was about a week ago too - very good turnaround.

Jeeves is easiest my second biggest referrer behind Google.

I just wish Yahoo would get its act together.

11:34 pm on July 12, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:July 12, 2004
posts:384
votes: 0


There was definately an update, traffic to some of my domains has increased by 10 times usual AJ levels.

New crawling has continued - normally updates occur on the 18th (ish) of every month according to my traffic logs.

11:50 am on July 16, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 19, 2004
posts:79
votes: 0


How is it possible to find out how many pages for a particular site are in the index?

Thanks,
Kev

Dayo_UK

9:27 pm on July 19, 2004 (gmt 0)

Inactive Member
Account Expired

 
 


Hi Kev

inurl:www.yourdomain.com

should do the trick.

Dayo

10:26 pm on July 19, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:July 30, 2003
posts:430
votes: 0


Last month AJ visited my new PR3 site and spidered 700 pages in one visit.

I haven't seen AJ since....

9:33 am on July 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


<back to reality>

The few hundred pages jeeves managed to finally find from one of my domains maneged to last all of nine whole days. Now POOF they are all gone again.

I guess Jeeves has all that cash on hand because they don't spend 50 cents to keep their hamster-powered search engine from constantly collapsing.

Checking several other sites with just one pfi page, it looks like they have completely removed all pages except the pfi ones.

10:08 am on July 20, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 1, 2003
posts:120
votes: 0


They definitely crawl like crazy. That is they consume about 40% of my traffic for last month and half at least. Average about 65000 hits a day.

And, wow, I get something like 0.04% of human traffic from them.

Gets on my nerves lately to tell you the truth. If there is someone from AJ/teoma here I'd love to hear them..

10:43 am on July 20, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 31, 2002
posts:51
votes: 0


site:www.domain.com domain seems to provide better results than inurl:www.domain.com - see the results for inurl:www.amazon.co.uk on ask.co.uk for example.

Ask Jeeves has been crawling a few of my sites heavily for the last few months, so much so I had to upgrade my hosting (1.3million hits so far this month on one site). But Ask Jeeves does now provide 33% of the volume of visits that Google does, so it has been worth it for me.

11:15 am on July 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


And now poof, pages are back in.

(I searched for text on a bunch of the these missing pages, not just the site and inurl commands.)

4:32 pm on July 20, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:June 22, 2001
posts:67
votes: 0


Teoma has been lighting us up for about 30,000 combined hits a day for a while now. While the traffic isn't as nice as Yahoo or MSN, it's about equal to Google. The bot is reasonably polite. Hard to turn it off when it's actually returning something.