homepage Welcome to WebmasterWorld Guest from 54.237.54.83
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
Inktomi crawls -- But doesn't index
Can anyone shed light on this?
Critter




msg:17037
 2:38 am on May 20, 2003 (gmt 0)

I've been crawled by Inktomi in March, April, and May...

Both March and April Ink got 3,000 or so pages each month.

So far this month I'm up to around 1,800 pages.

Number of pages in index (according to positiontech)...19.

This is really ridiculous: if Inktomi is going to crawl my site and eat up bandwidth then they should put those pages in its index...if not then don't frickin' crawl them.

Anyone else have this experience?

Peter

 

jdMorgan




msg:17038
 3:01 am on May 20, 2003 (gmt 0)

It just takes a long time...

If you don't want Ink crawling, add Slurp to your robots.txt Disallows.

Jim

Critter




msg:17039
 3:08 am on May 20, 2003 (gmt 0)

I don't mind Ink crawling...as long as eventually the pages get in there.

What's your experience with lead times?

Peter

jdMorgan




msg:17040
 3:39 am on May 20, 2003 (gmt 0)

I don't remember exactly - It felt like about four months!

It's been indexed solid ever since, though...

Jim

Critter




msg:17041
 2:56 pm on May 22, 2003 (gmt 0)

This is ridiculous...I checked positiontech this morning and the number of pages for my domain has gone from 19 to 12!

For three months straight Ink's gotten over 3,000 pages from my site each month (and is in there now), and they're taking my pages *out* of the index.

Can *anyone* shed light on this, or is Inktomi simply boneheaded?

Peter

martinibuster




msg:17042
 11:45 pm on May 22, 2003 (gmt 0)

Try your search this way, it might show more complete results:

domain:myname.com

Critter




msg:17043
 11:58 pm on May 22, 2003 (gmt 0)

That's what I did :)

Peter

percentages




msg:17044
 8:28 am on May 23, 2003 (gmt 0)

Critter,

This is a well known problem with Inktomi. Initially it will crawl all pages and show them in the index. Then it will delete the vast majority, step by step, until only two are left (can happen within a week). After that it will start adding them back again.

I have numerous sites with 30,000+ pages, they take time to remain stable in the index.

Inktomi is a roller coaster ride. First you get great results, then you fall into no-mans land. Then you climb the next hill very slowly, once at the top it doesn't kill you again!

Critter




msg:17045
 10:14 am on May 23, 2003 (gmt 0)

Heya %ages:

Thanks for the info! I suppose I'll just wait it out then.

(They shouldn't call them search engines...they should call them "patience tutors")

Peter

Dayo_UK




msg:17046
 12:54 pm on May 23, 2003 (gmt 0)

Excellent WW is full of gold!

Just about to post why have some of by pages in Ink disappeared and theres the answer above :)

nancyb




msg:17047
 3:43 am on May 24, 2003 (gmt 0)

:o I came here hoping to find a clue as to why one of my pages from before last October is still in Ink .... I'd be willing to trade if I could. Not only is the page gone, the whole site's been gone since 10/2002!

Dayo_UK




msg:17048
 7:38 am on Jun 13, 2003 (gmt 0)

Seeing a lot more pages in Inktomi from crawling activity end of May beginning of June :)

panic




msg:17049
 10:06 pm on Jun 16, 2003 (gmt 0)

A common reason for being crawled but not indexed by Inktomi is an editorial action.

From what I understand, Inktomi editorial checks pages at random to look for any "bad" content. That includes keyword stuffing, cloaking, and the worst of all, link exchange/link farms.

Critter




msg:17050
 10:28 pm on Jun 16, 2003 (gmt 0)

My site is squeaky clean.

Peter

frances




msg:17051
 10:10 am on Jun 17, 2003 (gmt 0)

Nothing to say to help.

Just wanted to tell Critter that my experience is the same as his. Ink crawls several times a week. Has done for months. NOTHING indexed yet.

SEO is here to teach us patience.

Kackle




msg:17052
 12:59 am on Jun 19, 2003 (gmt 0)

Inktomi is making me mad. I'm a nonprofit site that makes search engines look good, because 90 percent of my content does not exist in digital form in the known universe, apart from me. None of the stuff referred to below is PFI; I've never paid Inktomi a dime.

I had domain A and domain B. Domain A was the big one. Inktomi in early August suddenly showed about 50,000 pages and I was doing well.

Then I sold domain A and transferred everything to domain B, effective on January 1. From January 1 to April 30, the new owner of domain A, as per the contract, was doing a 301 for all my deep pages from A to B. By "deep" I mean everything but the index.html home page got redirected.

Inktomi continues to crawl my domain B regularly, often several thousand pages in a single day.

During the second week in April, Inktomi dropped almost all my pages on both A and B. Only about 2000 were left on each. Traffic dropped to almost zero. Traffic has been next to zero ever since the second week in April.

As per the contract, domain A stopped redirecting on May 13 (they were two weeks late). Ever since May 13, the new owners have been showing a 404 for my pages on domain A instead of redirecting them to B.

Today I noticed that 55,000 of my pages are now showing on one of the two domains in Inktomi. Guess what -- they're showing on the domain I sold, so they're all 404. My domain B is still next to zero!

Is this a sad story, or what?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved