doorpage and pain submission to Inktomi

[Cees]

Hi All,
I paid positiontech for home page and one other page which I like to use as doorpage.
Both pages were nicely picked up by positiontech and now listed as included.
But, I can't find the doorpage listed anywhere.
The doorpage is not link to on the home page. I don't really want to either ( not even invisible link).
I see Slurp/cat come in and grab both pages almost every day. A little while later I see Slurp/si come in at least several times a day grab the robots.txt and homepage but not doorpage.

ip based serving is still way out of my league. (I'm still learning and cloaking is too scary yet)
Any ideas other than having a link on homepage? (a previous post suggested that this may work)
Or perhaps a lesson in safe-proof/fool-proof cloacking?

Thank in advance
cees

[littleman]

Cees, is there anything interesting about your doorway page that may be setting off a spam alarm? Is your key word density very high, or is the doorway the product of a commercial software? Maybe you should try to put an unoptimized page in the doorway's place, just to see if that page gets indexed.

[Cees]

I just noticed that I made a nice typo, a 'paid' submission is becoming a 'pain' submission.

littleman,
I don't think there is anything special about the doorpage. It's very similar to one of my other pages, not-optimized. i figured that i could adjust/optimize the doorpage regularly because it should be included quickly after a change and that way i can tell whether the modification was an improvement or not. Call it research if you like.(worth $10)
I just don't want any other searchengine to pick it up because it looks so much like one of my regular pages.

[stcrim]

Cees,

There is really no go way to hide a page from the SE's as it relates to not linking it. Over in google's FAQ's they have a good scoop on why those pages are found anyway.

Also, my experience is - if position tech hasn't picked up that page, contact their customr service or sales dept. and they will take care of it...

good luck

[msgraph]

I had the same problem with Inktomi and the Slurp/si visiting the site all day. I posted some info on it in another discussion. Maybe you read it already but just in case...

[webmasterworld.com ]

[Cees]

srtcrim,
thanks for pointing out the not-linking bit.
The page in question is listed by positiontech as included and updated 12/04/00 (checked few minutes ago)
Slurp/cat is picking up the pages like clockwork.

msgraph,
I'm seeing the same pattern as you are describing, Slurp/si coming in several times in a row taking nothing but robots.txt and homepage.

I'll put a link to the seconpage on homepage today, see if now wil be able to find it on iwon/canad/msn etc.

But, what if you have a real case of wanting a page in Inktomi which you don't link to on the root/index page?

Has anyone submitted just a page that isn't root/index page?
Did it get listed promptly?

[tigger]

Cees

I submitted doorways as a test to PT some time ago they were all indexed within 3/7 days by ink and are on the various partners (some have good rankings), not that I see much for it, although I have noticed old index pages coming up as well that have not been submitted for a long time

[Cees]

tigger,
those doorpages, were they linked to by your root/index page?
I'm puzzeled by Slurp/si coming in several times a day and grabbing nothing but root and robots.txt.

I don't know, this Inktomi is to dificult for me. Even for the homepage on e.g. search.msn.com it is a case of 'now you see it, now you don't.'

[Cees]

Guess what,
I just check on iwon.com and both pages are now in the database.
Checked the log files and Slurp/cat picked up both. Several hours later Slurp/si picked up root and robots.tx (root now has link to doorpage) and voila I'm in iwon.com. Do they work that fast or is this just coincidence?
I'll take link off again, see what happens.
Will it stay or willit go?
Cees

[Brett_Tabke]

It will go Cess...eventually. Ink is historically the slowest at removing pages that are added to robots.txt. I've had pages last for (cough) a year after adding them to a robots.txt.

[Cees]

Hi Brett,
Perhaps you misunderstood. The doorpage is not listed in robots.txt I did not have it linked to on my homepage. Only after I added the link on homepage I saw it included but this happened within a 24h time period, perhaps coincidence.

You gave me another idea though. Would this work for robots.txt (assuming it's obeyed)

User-agent: Slurp
Disallow:

User-agent: *
Disallow: /door.html

[websurfer]

Greetings and happ holidays. Regarding Robot.txt does anyone feel that they should be a mandatory inclusion? I have never used them and have had some really good results. However, as we all do, I am looking do achieve better results and gain all the advatage I can. All suggestions are greatly appreciated, thanks.

[Cees]

Verdict is in. After a few weeks of testing I'm coming to conclusion that if you have homepage and doorpage crawled by positiontech than you need a link on the homepge to the doorpage. If you don't than positiotech will report it as included by page will not show up in the database.

Anyone with similar/different experience? What to do. I don't want link to the second page on home page. What if you only submit page which is not homepage?