Ink 'free' spidering gone dormant?

Slurp hasn't grabbed much lately....

[mivox]

What's up with Ink's "regular" spider activites? I'm not talking about free submit, as I haven't submitted my pages for a while, and I don't have any paid pages either. But their fairly regular crawls have disappeared...

The last month or so, all Slurp ever requests is my root directory. Anyone know what's up?

[nicebloke]

I've been slurped maybe 20 times in the last month.

[mivox]

Did they request actual html pages when they visited?

Ink visits my site almost daily, often twice a day, but they ONLY request my root directory. I don't know how long it's been since Slurp requested an actual html file...

[oilman]

I got slurped out of nowhere on a brand new domain last night by 8 different slurp IPs but nothing beyond the root.

[nicebloke]

sorry yeah they've all been the root.

[msgraph]

I've seen them deep-crawl about 30 sites for the past month. They've been in those sites every single day grabbing tons of pages using both.....

Mozilla/3.0 (Slurp/cat; [e-mail]; [inktomi.com...]

and

Mozilla/3.0 (Slurp.so/1.0; [e-mail]; [inktomi.com...]

[pete]

MsGraph, same here.

I submitted four pages on the 24,25,29 of Jan and 2 of Feb to the Ink engines of a 40 page site.

The pages are now listed in Hotbot and Canada

[Machiavelli]

Root only up to 5 times a day per domain for me.

[starec]

The same "root only" experience here. Maybe this regular brief slurp-touch is to remind us that if we paid things would look different.
No pay, no slurp...

[mivox]

> brief slurp-touch

LOL... somehow, that doesn't sound like anything I want to be involved in! *eww*

Maybe if I try resubbing a few changed pages to Canada or someone they'll come take a closer look.

[Brett_Tabke]

This may get me tarred and feathered by the community, but the trick is to submit any pages linking to you before submitting your site. Wait thirty days inbetween those to acts.

[mivox]

I've submitted ALL the pages I have linking to me... over two months ago for most of them.

All my pages *used* to be listed in Ink, but after their last update, my index page was the only page left showing the the SERPs for most of their partners. (Excite seems to have included TWO of our page in the last update tho') But, most of the pages linking to us are still in their DB/SERPs.

I'm just trying to think of some way to get Slurp to do a decent crawl of the site, and hopefully some of the pages will re-appear next update...

[ihelpyou]

By George!..... now I know why I'm not linking to Brett. :)

[mivox]

So... any non-paid sites see any thorough Slurp spidering in the past month? I've still only gotten a teeny visit here or there, and I still only have one (now slightly out of date) index page in their DB...

[jimmykav]

Hi mivox
i have a non-paid site and slurps been all over my site like a bad suit as usual.
regular visits in small doses.

[mivox]

regular visits in small doses

That's what I get... the odd part is that none of the pages it visits one-or-two at a time ever seem to appear int he DB. It has, however, progressed beyond *only* requesting the root directory.

Maybe we'll be seeing a change in the listings next time they do a big update?

[Marcia]

I have seen much slurp activity for a site with nothing paid. BUT for a site with one paid page (the index page) I hve not seen anything from slurp at all, whatsoever, this entire month. 48 hours? Not judging by me, it isn't.

[Xoc]

Slurp is visiting fairly frequently, downloading robots.txt and the home page, then goes away. They have not even updated the index to reflect the changed home page since at least January. Even Excite is more current than they are now!

[2_much]

Very heavy activity from Ink for us, on both paid and non paid sites.

Anyone else noticing a certain degree of consistency between google and inktomi (if the sites are paid for in inktomi)?

[oLeon]

>>Slurp is visiting fairly frequently, downloading robots.txt and the home page, then goes away

I can easily confirm this: the same on my site!
Slurp comes up to four times a day to download the robots.txt, and now and then the homepage - almost every day!
Then after playing this game one month, one of the hidden spider from INK came across and checked the subpages (just a few not all), two times within a week.
But nothing happened after that.
Why does Slurp crawl that much without indexing? No sense...

[tigger]

you may well find the site is indexed but perhaps burried. I see Slurp all the time within the logs of my clients but the only pages that are ranking well are the PT ones and maybe the odd index but that's very far & few between :(

[engine]

I'm also pretty certain that the permanent (non-paid) db gets different treatment. More regular visits.

[tigger]

Have you also noticed the lack of paid pages if you do a search using one of inks partners, I would have thought by now more pages would be showing but it still seems some of the old database pages are still out ranking the paid submission.

[mivox]

you may well find the site is indexed but perhaps burried

I search by domain and business name to confirm whether I'm listed or not... my index page shows up on all Ink partners, and 1 secondary page also shows on Excite. That's it. I haven't given them a penny, and I think they're miffed.

[Marcia]

>you may well find the site is indexed but perhaps burried

I can't tell. I just had a new page added, that I know, but MSN only goes up to 200. ;)

[tigger]

Marcia

are you trying to see if your site's in the inktomi database?.

Try using canada.com I find that a good route for true returns from inktomi & also try searching for the url on canada

[oLeon]

The page/site isnīt buried. Itīs not in the index for sure.

BUT -
I got another page into the index of INK without paying. I am not sure how it happened, but Iīd assume that slurp followed a link from a directory-listing (ODP or something else).

(BTW: the same on this page, over and over spidered, almost every day, and also a deep crawl, but only the rootpage got into the index. All that happened in dec 2000 - the site is online since nov 2000.)

[drbill]

Being Spidered and deep crawled for the last 3 weeks.

These are pages that I have never submitted. only a few from the 1st week showing in the database...

[stcrim]

These days Slurp is a pest that clutters my logs with useless evidence that he has come by. I wish INKTOMI would quite sending him around. It's a waste of my time (and Log space)

Any question about how I feel about INK these days...

-s-

Note: If INK stops their "will spam for cash" program and starts to farm the net for real search results again, I would change my tune.

[msgraph]

If it's worth anything check the last post I made in this forum.

[webmasterworld.com...]