Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Inktomi crawls -- But doesn't index

Can anyone shed light on this?



2:38 am on May 20, 2003 (gmt 0)

10+ Year Member

I've been crawled by Inktomi in March, April, and May...

Both March and April Ink got 3,000 or so pages each month.

So far this month I'm up to around 1,800 pages.

Number of pages in index (according to positiontech)...19.

This is really ridiculous: if Inktomi is going to crawl my site and eat up bandwidth then they should put those pages in its index...if not then don't frickin' crawl them.

Anyone else have this experience?



3:01 am on May 20, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

It just takes a long time...

If you don't want Ink crawling, add Slurp to your robots.txt Disallows.



3:08 am on May 20, 2003 (gmt 0)

10+ Year Member

I don't mind Ink crawling...as long as eventually the pages get in there.

What's your experience with lead times?



3:39 am on May 20, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

I don't remember exactly - It felt like about four months!

It's been indexed solid ever since, though...



2:56 pm on May 22, 2003 (gmt 0)

10+ Year Member

This is ridiculous...I checked positiontech this morning and the number of pages for my domain has gone from 19 to 12!

For three months straight Ink's gotten over 3,000 pages from my site each month (and is in there now), and they're taking my pages *out* of the index.

Can *anyone* shed light on this, or is Inktomi simply boneheaded?



11:45 pm on May 22, 2003 (gmt 0)

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Try your search this way, it might show more complete results:



11:58 pm on May 22, 2003 (gmt 0)

10+ Year Member

That's what I did :)



8:28 am on May 23, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member


This is a well known problem with Inktomi. Initially it will crawl all pages and show them in the index. Then it will delete the vast majority, step by step, until only two are left (can happen within a week). After that it will start adding them back again.

I have numerous sites with 30,000+ pages, they take time to remain stable in the index.

Inktomi is a roller coaster ride. First you get great results, then you fall into no-mans land. Then you climb the next hill very slowly, once at the top it doesn't kill you again!


10:14 am on May 23, 2003 (gmt 0)

10+ Year Member

Heya %ages:

Thanks for the info! I suppose I'll just wait it out then.

(They shouldn't call them search engines...they should call them "patience tutors")



12:54 pm on May 23, 2003 (gmt 0)

Excellent WW is full of gold!

Just about to post why have some of by pages in Ink disappeared and theres the answer above :)


3:43 am on May 24, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

:o I came here hoping to find a clue as to why one of my pages from before last October is still in Ink .... I'd be willing to trade if I could. Not only is the page gone, the whole site's been gone since 10/2002!


7:38 am on Jun 13, 2003 (gmt 0)

Seeing a lot more pages in Inktomi from crawling activity end of May beginning of June :)


10:06 pm on Jun 16, 2003 (gmt 0)

10+ Year Member

A common reason for being crawled but not indexed by Inktomi is an editorial action.

From what I understand, Inktomi editorial checks pages at random to look for any "bad" content. That includes keyword stuffing, cloaking, and the worst of all, link exchange/link farms.


10:28 pm on Jun 16, 2003 (gmt 0)

10+ Year Member

My site is squeaky clean.



10:10 am on Jun 17, 2003 (gmt 0)

10+ Year Member

Nothing to say to help.

Just wanted to tell Critter that my experience is the same as his. Ink crawls several times a week. Has done for months. NOTHING indexed yet.

SEO is here to teach us patience.


12:59 am on Jun 19, 2003 (gmt 0)

Inktomi is making me mad. I'm a nonprofit site that makes search engines look good, because 90 percent of my content does not exist in digital form in the known universe, apart from me. None of the stuff referred to below is PFI; I've never paid Inktomi a dime.

I had domain A and domain B. Domain A was the big one. Inktomi in early August suddenly showed about 50,000 pages and I was doing well.

Then I sold domain A and transferred everything to domain B, effective on January 1. From January 1 to April 30, the new owner of domain A, as per the contract, was doing a 301 for all my deep pages from A to B. By "deep" I mean everything but the index.html home page got redirected.

Inktomi continues to crawl my domain B regularly, often several thousand pages in a single day.

During the second week in April, Inktomi dropped almost all my pages on both A and B. Only about 2000 were left on each. Traffic dropped to almost zero. Traffic has been next to zero ever since the second week in April.

As per the contract, domain A stopped redirecting on May 13 (they were two weeks late). Ever since May 13, the new owners have been showing a 404 for my pages on domain A instead of redirecting them to B.

Today I noticed that 55,000 of my pages are now showing on one of the two domains in Inktomi. Guess what -- they're showing on the domain I sold, so they're all 404. My domain B is still next to zero!

Is this a sad story, or what?


Featured Threads

Hot Threads This Week

Hot Threads This Month