Welcome to WebmasterWorld Guest from 54.167.252.62

Forum Moderators: open

Message Too Old, No Replies

Inktomi spidering non-paying websites

     
11:57 am on Dec 29, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It seems that Slurp over the last few days has begun deepcrawling my websites. I have certainly never paid anything for that.

New politics from the new owners? I have been visited before but always with only one hit.

12:05 pm on Dec 29, 2002 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member



It not unusal to see Slurp in your logfiles. Unfortunatly, this does not mean that your site will be included in the index.

I care for many sites wich are still crawled by Slurp on a regular basis but where never included in the index for free.

12:34 pm on Dec 29, 2002 (gmt 0)

10+ Year Member



why should they crawl without indexing a site?

my site has been deepcrawled too (since wednesday) - i didn't subscribe to basic submit, but i have 3 listings in odp.

12:38 pm on Dec 29, 2002 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member



In fact Ink has indexed sites all the time without being paid, how else could they have collected such an enormous db?
I have never paid Ink for any site, yet eventually they all got in.
12:38 pm on Dec 29, 2002 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member



>why should they crawl without indexing a site?

Maybe to let you know they are still alive. :)

Joke aside, I believe you have more chances to get a free ride if the site in non English or in niche markets.

12:45 pm on Dec 29, 2002 (gmt 0)

10+ Year Member



I agree heini!
yes macguru: the site is in german :)

similarly to troels my site has got 2-5 hits per month since september, but since wednesday 250 hits and 6.5 MB. so I assume I will get indexed soon.

btw: what does basic submit with the url's mean? if I subscribe *one url*, will my whole site get indexed or just *one page*? (would be obviously very expensive!)

12:48 pm on Dec 29, 2002 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member



It's 39 $ for the first page and 25 $ therafter for every single page you want crawled.
12:58 pm on Dec 29, 2002 (gmt 0)

10+ Year Member



>It's 39 $ for the first page and 25 $ therafter
>for every single page you want crawled.

that's ridiculous, I have currently about 400 sites and I would never pay so much money (10039$?) just for getting indexed! that's thousands of clicks i could get from google-adwords.

i hope msn will soon switch to FAST...

7:17 pm on Dec 29, 2002 (gmt 0)

10+ Year Member



I have submitted some sites to Inktomi but most of mine got crawled over time. IMHO its real simple, you pay for quick inclusion and monthly guarenteed visits.
8:17 pm on Dec 29, 2002 (gmt 0)

10+ Year Member



I have several sites that I have not paid INK for inclusion, they are all crawled regularly and included in the index - some have great ranking too.
9:17 pm on Dec 29, 2002 (gmt 0)

10+ Year Member



>I have several sites that I have not paid INK for
>inclusion, they are all crawled regularly and included in
>the index - some have great ranking too.

how did you submit them and how long did it take them to show up in the results?

7:09 pm on Dec 31, 2002 (gmt 0)

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Ink has been hitting me hard for the last week, after only having a few pages in. They aren't hitting me as hard as Google during the main crawl, but they have been averaging over 300 pages a day, and they are going deep.

I wonder if Yahoo told them to index more non-paying sites.

<added>I never paid, and never submitted my site, as I wasn't going to go to the effort of figuring out how to submit without paying</added>

10:19 am on Jan 16, 2003 (gmt 0)

WebmasterWorld Senior Member chiyo is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I really do doubt that if Y! seriously wanted to get people coming back to their search that they would limit the ink listings in Y! to paid pages only. That would be the quickest way to send people back to G! and maybe trying out ATW/T/Wise/ etc.

However i think the way they list ink database listings will be complex and clever, maybe providing some sort of advantage to paid sites, without reducing the functionality and power of their present search using Google.

I think its possible that the reason Y! bought Ink was so they could spider the web themselves and produce their own databases and ranking algos rather than relying on Google. Its all about contol, and G! surely would not have provided enough flexibility in the use of their results and their spiders.

I still feel strongly that they bought Ink for their search technology rather than their database or listings. Using the spidering tech would be part of that.

Personally im not worried at all about opting for ink until we know how Y! will merge their new acquisition. But obviously you should be in there as a definitely! - paid or not. Thats about all im worried about.

So the present spidering splurge is not a great issue for me. To me the significance now is just to remember its Y! spidering you now, not ink, which used to spider on behalf of a lot of customers, unless Y! continue to lease/sell their results to second parties like Ink used to.

11:23 am on Jan 28, 2003 (gmt 0)

10+ Year Member



The very same question for crash:

how did you submit them and how long did it take them to show up in the results?

>I have several sites that I have not paid INK for
>inclusion, they are all crawled regularly and included in
>the index - some have great ranking too.

Tor

11:37 am on Jan 28, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



how did you submit them and how long did it take them to show up in the results?

I, for my part, submitted:

here
[submitit.bcentral.com]

11:50 am on Jan 28, 2003 (gmt 0)

10+ Year Member



So Microsoft is not completely EVIL :)
Anyhow, how long does it take to get indexed after submitting there?

Tor

12:07 pm on Jan 28, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It takes some months normally, but it is worth it because we have many sites that we are responsible for. PFI would have cost us big bucks. ;)
12:17 pm on Jan 28, 2003 (gmt 0)

10+ Year Member



some means how many? 2,3,4,5,6...?

Tor

12:18 pm on Jan 28, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



3-4
12:35 pm on Jan 28, 2003 (gmt 0)

10+ Year Member



thank you
12:59 pm on Jan 28, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I might add that in january Inktomi have been extensively crawling my websites. They are easily #2 before Fast. Anybody seing the same pattern?

Tor

1:04 pm on Jan 28, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Heavy crawling of all our sites from Slurp in January.
1:26 pm on Jan 28, 2003 (gmt 0)

10+ Year Member



What's the inclusion time for free addurl in fast?

Tor

1:42 pm on Jan 28, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I would say 3-5 weeks.
2:40 pm on Jan 28, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Its probably worth mentioning that some experts who post here at WebmasterWorld believe that submitting to Inktomi is a mistake. The theory is that submitting sends you to the bottom of the barrel as far as rankings are concerned.

The better approach for free listings, so the belief goes, is just to let Slurp find you.

2:47 pm on Jan 28, 2003 (gmt 0)

10+ Year Member



so, how often does slurp crawl a site?
2:51 pm on Jan 28, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Well, from all I can see, Slurp is crawling constantly.
3:01 pm on Jan 28, 2003 (gmt 0)

10+ Year Member



why should they crawl without indexing a site?

Link pop is definitely part of the Inktomi algo, so it helps if they can crawl lots of pages but they don't necessarily have to index them all.

3:01 pm on Jan 28, 2003 (gmt 0)

10+ Year Member



what does it mean? every day? hour? month? second?
8:50 am on Jan 29, 2003 (gmt 0)

10+ Year Member



Going back to bcentral (MSN) submission to Inktomi (mentioned above).

Can you only submit the Domain Name or can you submit as many pages as you like.

Many thanks

This 38 message thread spans 2 pages: 38
 

Featured Threads

Hot Threads This Week

Hot Threads This Month