How often is Ink updating now?

[born2drv]

Are they monthly like Google? Or do they do parial crawls more frequently? Any preferences to crawl sites that have higher link popularity?

Just wondering when I should expect my site to be re-indexed again, I would like it to go a bit deeper :)

Thanks.

[thejenn]

The paid inclusion urls are updated every 48 hours.

The entire index appears to be updated about once a month, though I remember just reading an interview with someone from Ink, and they stated that they were trying to get it down to every 18 days or so. (The article discussed a strategy of aiming for frequency of updates rather than sheer number of sites in the index....)

[Inktomi]

That's correct, our paid inclusion sites are updated every 48 hours and the entire index is now updated about once every two weeks.

Inktomi.

[PaulPaul]

What about non-paying inclusions??

[Nick_W]

As inktomi said, once every couple of weeks...

Nick

[c1bernaught]

How often is a site in Ink spidered?

[Grumpus]

I don't think it really does a "crawl" in the way that Google does. I get 20 or so hits a day from it - every day. Sometimes it's the same pages, sometimes it's new and different ones. Some make it into the index, some don't.

Not sure how it finds the pages, either, as it's not soley DMOZ listings - though those tend to get crawled pretty regularly. There are other odd pages that I think it must find one on my site that links to it one day, then adds it to a queue, and then gets around to it several days later? I dunno.

G.

[PaulPaul]

As inktomi said, once every couple of weeks...
Nick

Sorry, I misunderstood, I thought Ink meant that paid sites were spidered every 2 days and updated once every couple of weeks.

[c1bernaught]

Grumpus,

O.K., lets say your site has ... 3000 pages. Are you saying that INK will probably never crawl more than a few a day? That would mean that content is not king. A webmaster would then need to pack either the index page, or the INK included page, with all of his/her primary and secondary keywords. Hmmm...

[Grumpus]

I don't know that it "never" will - well, actually, I do - it won't. I've got over a million pages. What it means is that I don't know what determines which pages get crawled and which get put into the index. It's fairly obvious that at least one of their seeds is the DMOZ, but beyond that, I have no clue. Content doesn't mean much, either. I've got pages with little or no information on them (i.e. Such-and-so played "Webmaster #1" in "Hackers" or something where there just ISN'T info) that have made it into the index, and then there are pages that are popular, are indexed in several directories and search engines, and HAVE been crawled by INK but just don't appear. <shrug>

It's a mystery to me. I can say I've never seen it enter on a page and start crawling links it comes across like Googlebot does.

G.

[Slud]

If Google "Dances" (ugh). Inktomi "turns over the compost heap".

[chowcat]

I've been crawled everyday by Inktomi Slurp almost ever since I paid the $30 to ineedhits. I've also noticed that MSN/Inktomi is starting to throw out much more relevant and better SERPS than ever before. It seems in many respects similar to Googles algos, but without the page rank "bite".

[c1bernaught]

chowcat,

How many slurp hits are you seeing each day?

[McMohan]

Everyone here is talking about spidering non-paid sites. Are these the sites included in Inktomi's database before it went paid or the sites included in its db by inktomi itself? Coz, to my knowledge Inktomi doesn't have free inclusion option.

Mc

[Macphisto]

I never paid for inclusion and my site i also crawled everyday. I logged between 3 and 194 hits for the last 30 days. Average for 30 days is 61 sites per day. Overall 10% more hits than google since i started this agent log.

Matt

[Grumpus]

McMohan - as with most of the spiders out there, pay or not, some pages from your site are likely to get into them for free - eventually. Links from sites that do pay can do it. For INK a DMOZ listing seems to help. The thing is that they don't tend to get "updated" very often. (Heck, I've got pages in AV from April). It does feel like INK is updating more frequently though, but it's only been about 3 weeks since my site's glitch was removed. (When I paid, my free pages vanished, they're back now thanks to Inktomi(guy)'s involvement here.) It's hard with only 3 weeks of information to work with to tell what's getting updated and what is not.

I guess I'll know more in a few more updates (or another month).

G.

[Marcia]

McMohan, Inktomi does have free submit. Scroll down, it's there:

[hotbot.lycos.co.uk...]

[chowcat]

Sorry for the delay "c1bernaught"...

about 15 hits from Inktomi Slurp on a daily basis. Not a huge amount, but more than any other spider.

[c1bernaught]

chowcat,

Do your stats actually say "Slurp"? I don't see "Slurp" anywhere in my stats.

[diddlydazz]

I am still waiting on URLs to be included, submitted them last week but they are still not in the index, just "queued for inktomi".

I was under the impression that inclusion takes 48hrs or so i was lead to believe.

Looks to me like Inktomi need to sort things out, or at least have a word with ineedhits, it would be nice if ineedhits answered the email that I sent.

Anyone had similar experiences ?

By the way the URLs are not dodgy in the slightest, no cloaking, no hidden text, in fact they are hardly optimised.

Dazz

[rfgdxm1]

Agent: Mozilla/5.0 (Slurp/cat; [e-mail]; [inktomi.com...]

[McMohan]

Marcia,

Whether submitting in [hotbot.lycos.co.uk...] gets ones' site indexed in only the .uk Search Engine or in the main db? Thnx for the info though, quite valuable.

[c1bernaught]

One of the biggest issues I've had with INK is the "48 Hour" update. In my experience the update takes at least 72 hours and sometimes more. It has not been uncommon for it to take 96 hours to see changes made I've made to my site. However, they get right on it if you send them an email.

[przero2]

When is the update awaited?. We had a site completely (almost except for 0.01% of the pages) removed almost 10 days ago. Since then Slurp crawled this site and hoping to see a return of this site in the core Inktomi DB (I mean most pages and not just 0.01% of pages)

[gsmitchell]

Dazz,
I too have had the same problem with ineedhits when it comes to them answering their emails. As for Inktomi, I know that ineedhits was having issues updating and even if your site was showing still queued after 48hrs your site was submitted. If you do a search for your site I bet you it is there. Since submitting my url's last month I notice that the refreshes have been taking 48-72 hours

[Spearmaster]

Inktomi -

I have PositionTech for indexing. I get "Disallowed by robots.txt" - problem is, I don't have a robots.txt (and when I did it used to work just fine, since it's the Webmaster World list).

Is it possible that Slurp! is being blocked by .htaccess? And if so, are you using a new agent which identifies itself as something other than Slurp!? None of the IPs I have blocked appear to match or even come close to yours.