why does inktomi crawl us but our pages don't show up?

Forum Moderators: open

Message Too Old, No Replies

why does inktomi crawl us but our pages don't show up?

sm26

6:53 pm on Jul 25, 2001 (gmt 0)

hi everyone,
my web site gets about 8k pages crawled every day for months now and i still don't see my pages at hotbot.
1. is ink just really slow and processing pages?
2. have i been banned?
3. they're doing this for research? see [searchengineworld.com...]
4. any other ideas?
thanks
rich

stcrim

7:13 pm on Jul 25, 2001 (gmt 0)

sm26,

Welcome to WMW and the INk forum. They will gladly crawl you for free - but to get indexed in INK now costs handsomely $$$$$$$. Kind of reminds me of the oldest profession - the teasing is free...

-s-

Travoli

7:58 pm on Jul 25, 2001 (gmt 0)

Oh, that is interesting. We have not paid, but we still get hit by Slurp hundreds of times daily.
So we are not included in the index at all? We are listed with the partner sites. Too bad there is no direct wayt to know. Or is there?

sm26

5:58 pm on Jul 26, 2001 (gmt 0)

thanks stcrim!

even though ink now has index connect and a host of resellers, why would they bother wasting resources crawling our site if we're currently not paying? no one at ink has contacted me about the crawling activity and paying to get indexed.

moreover, their index has pretty recent content from small/individual content sites that probably can't afford to get indexed.

i'm hoping that it's a matter of time before we start showing up.

stcrim

9:23 pm on Jul 26, 2001 (gmt 0)

Some informational sites may very well be getting indexed - but the minute a page even hints at being optimized I wouldn't count on it.

Good luck

-s-

ps201

12:56 am on Jul 30, 2001 (gmt 0)

stcrim;
I see the same activity from Inktomi. What is your expert opinion on delay between pages being crawled and them showing up on msn and hotbot? (for sites that are not oaying)?

They are crawling a lot of pages from my site but have not seen the pages on hotbot or msn. Why would they crawl so many pages -- if they had no intention of showing them in the search results.

I do not even see the point of "teasing" as you mention. If anything, crawling pages would get any potential paying customer to wait and see instead of going to them and agreeing to pay for indexing.

stcrim

2:14 am on Jul 30, 2001 (gmt 0)

PS201,

Welcome to WMW and The INK forum.

It may simply be a matter of keeping up with the web in case they have to provide results that are not skewed by profit. Or it may be they have clients that want the information - to be honest I don't really know. Seems like a big waste of resources to me...

-s-

ps201

6:15 am on Jul 30, 2001 (gmt 0)

thanks stcrim

caine

1:05 pm on Aug 1, 2001 (gmt 0)

I have the base url page listed and that is it, and Ink has successively crawed for the last 3-4 months, though non of the other pages have been listed. Also July's crawl was the largest from INK, so i am patiently waiting as i don't want to pay PT or any other ransom, if it is possible to get listed for free.