Inktomi Update Schedule

Slurp finally deep-crawled my site. Now what?

[jimh009]

Slurp has finally taken in a good part of my site (the free way) after many, many months of just grabbing the home page and going. Maybe the whole Yahoo thing has something to do with it?

Anyway, was wondering how long it takes from the time a site is crawled by Slurp to when the site starts showing up in their SERPS? Is Inktomi like Google, with a monthly update cycle? Or does it take longer or is the time period shorter?

Also, once a site is crawled by Slurp, will slurp come back on a more or less monthly schedule like Googlebot? Or is the arrival of Slurp kind of a crap-shoot?

Thanks.

Jim

[HuhuFruFru]

i was wondering the same questions as you, jim009:

my site was deepcrawled yesterday too (without paying 25$, but waiting for 2-3 months)

[kevinpate]

My take on Slurp is it's a crapshoot when it rolls around, but once Slurp does roll the dice, it tends to return to your casino shouting 'c'mon now, baby needs a new pair of shoes" for several days in a row.

[Thanasus]

Just because their bot visited you doesnt mean you'll get included. I don't know why they would waste resources visiting pages that they don't index, and do it over and over, but it happens. Many of my pages got slurped every month over the past 12 months but aren't indexed. Some get indexed within one month. I have yet to figure it out.

[Brad]

It used to matter about how many incoming links to you that Ink could find - usually over a couple of months or more. If they found enough good links and perhaps other criteria sometimes you would make it in. Perhaps that is still part of it. Incoming links cannot hurt in the process, either on Ink or other engines.

[silinx]

Y'know, Slurp has hit my site a TON (I think they've hit every page) in the past two weeks, but, I haven't seen any changes in the results.

Also strange, they're showing my UK Yahoo! directory listing but not my regular old Yahoo! listing.

[peterdaly]

I am hoping I start to get some traffic from the inktomi engines, almost all of my traffic comes from Google right now.

I just made a huge dynamic expansion on my site, and inktomi has been crawling it for 5 days now, although it looks like it fininshed this morning. I keep getting crawler hits, but not like I have been for the past few days.

If I don't get traffic from it, I have to block it next time it comes around due to bandwidth usage.

Should I be making optimizations to my pages for inktomi? What does it care about? Where can I find and up to date document about inktomi, from a webmasters point of view?

Thanks,

-Pete

[HuhuFruFru]

you want to block inktomi? wow, you must be crazy...

wherever you live or whatever your business is- how do you know that inktomi won't become the 2nd most important robot after the yahoo-takeover?

[peterdaly]

HuhuFruFru,
Inktomi has used up more bandwidth in the last week then all of my human visitors combined. I would not be suprised if inktomi over the past 5 days used up more bandwidth than humans over the last month or more. (I have yet to run detailed reports)

I currently have effectivly ZERO traffic from any search engine but Google. If after a appropriate amount of time I don't have Inktomi based traffic, it would make no sense to waste the resources on Inktomi's crawl, I would restrict it just to my index page. If in the future it becomes more valuable, I would have to reevaluate the decision.

That said, I am not sure Inktomi has ever deep crawled me before, so I have no idea what the results are going to be. I for certain have never had enough pages before to keep it busy for 5 days of constant crawling. Earlier in December when Google last crawled me, it picked up about 2000 pages, and I know I now have many many times that. I am waiting on pins and needles for the next Google update (Dec?), seeing how in the Nov. update I had 17 pages indexed.

I will not shut inkomi off if traffic comes about from it, but right now it may just be a waste of bandwidth. We will see.

-Pete

[korkus2000]

Welcome to WebmasterWorld peterdaly,

To perform well with Ink, you need to target phrases that don't fill up with Looksmart and Overture results. Ink is a different beast than Google. You need to optimize for it or you really won't perform well with it. You may not even be able to do it because of your market. I would be hesitant to block it though because you may not have a free way in in the future.

Ink seems to be doing 20 - 30 day updates. It never really hits the radar here because the traffic gap between Ink and Google is so large. If you are paying Ink then the update is daily.

[gsmitchell]

Koorkus,

I thought that if you were paying INK the updates were every 48 hours and not every 24 hours. Am I missing something? I use INeedhits for my paid inclusion

[korkus2000]

No your not missing anything. Its like 48 hours. I am just saying if you pay you are constantly being updated.