A local crawler, feeding searchengines with fresh content from your site?

[maraeger]

Have any of you had any experience with this: linkloader ?
I short, it seems to be a locally controlled crawler, able to feed searchengines with updated and added pages from your site.
I guess this isn't one of the regular "submit-to-more-than-1000-searchengine's" programs... ;)
But does this actually work?
It also seems like it's a free service, at least for now.
I haven't tried it myself, (will sign up later today), but I'm just curious to hear if this is a system you guys use, and if so: do you get any good results?

They have also posted a link to some sort of article: www.hegnar.no [hegnar.no]
Does anybody feel like translating any of this article's content? (I think it is in Norwegian)

(I'm not sure if this is the right place to post this, so moderators, feel free to move this to a more appropriate group)

[edited by: Brett_Tabke at 10:56 am (utc) on July 4, 2002]
[edit reason] shortened side scroller url [/edit]

[Brett_Tabke]

I believe that in order to do what they claim, they would have to have agreements with the search engines. I have not heard of any such agreements. I think it's a submission program and carries the same risks to your site.

[maraeger]

Hi, Brett!
Thanks for the quick response.
I've gone over their site again, this time a little more thorough.
I can't find any references to searchengines they could have such agreements with, either.
Still, the ting that caught my interest, and what sets this project apart from others I've seen, is their claim to deliever a fully controllable crawler capable of delievering just the updated parts of a site.
If they are able to deliever this data, it would surely be a sinch for SE's to index it as fast as it's delievered...
My initial thought was that this would be a very good idea.
But ofcourse, as you say, this would require them having agreements with the searchengines in question.

[maraeger]

Update:
Yesterday, I registered and downloaded their client.
It seems to be very configurable, and runs without any problems here on my Win2K box.
I've set the crawler to run 2 times 'a day, and transmit the changes made to the site it crawls; which it does.
I can also log on to my account at their site, and access at my last transmissions and see how much the site has changed since the last crawl.

Beeing concerned with how they actually are going to deliever these results to SE's, I sent them a mail asking if they indeed have such agreements with SE's.
I got a mail back early today, in which they are stating that they currently are in talks with several major SE's.
They are also saying that until they have any formal deals with SE's, the service will remain free, and the data they receive wil be used in test-trasmissions of URL's with several SE's.

I think this system has great potential, and hope they're successful in landing deals with SE's.

[chiyo]

Am I reading this right?

1. Some guys are telling us they have a system for notifying search engines of your updates

2. They say the service is free

3. When asked by email how it works they say it dosent work (as advertised) yet. Also they havent got any agreements with Search Engines. So we are basically beta testers for them and probably providing them with some nice juicy data for free.

4. However, when it DOES work and they DO have agreements with Search Engines, it may not be free anymore. But they promise it will be free until it starts working. Hmm..

Conclusion: This is a free system until it works. When it works we will charge you for doing what the Se's should be doing themselves or charging directly for without a third party. (like Ink etc)

The crux is - Why would search engines want a third party way to find fresh pages when google, ATW, and many others do it already at no cost to us?

And finally why could this be a useful service when we are paying a third party to do something that Search engines do for free anyway.

Do the makers of this software seriously think the SE's need help in finding fresh or updated pages and are willing to pay somebody else to send them stop press fresh new spam? Really I thought Se's have gone beyond the stage where they take page/site submissions seriously. They have already told us that 95% is spam. The smart ones follow links from respected sites.

Is that the way other's read it? Let me know 'cause Im scratching my head here trying to understand whether its a curious scam of putting themselves between SE's and ourselves and charging us for the privelage or is it just plain fuzzy headedness?

The only "potential" I see this system having is for the creator if he finds enough innocents.

[maraeger]

Hi chiyo.

This is how I understand it:

1. Yes

2. Well, you can download their client for free.
They're not saying anything about what will remain free and what will be a paid service.
I guess the system is in its testing stages, and therefore is free to the public...

3. They don't, (obviously), say that their system doesnt work.
But it's hard for me to se that they can guarantee any indexing by SE's without having any agreements with them (again; pretty self-evident).
What they do state, is that they're in talks with SE's.

4. See 2.
They do not give detailed information on how payment will be implemented in their solution.
The way they explain how their system works, it seems to be some sort of a paid-inclusion service.

The benefits of this system, as I understand it, is that you're able to immedeatly transmit your updated content's URL's to SE's without having to wait for a spider to come along and crawl your whole site.

This way the SE's doesen't have to waste resource's_finding_ the updated/new pages (an upside to them).
And the site would have it's updated/new pages more quickly indexed (at the default settings in their crawler, I can transmit updated content in 12 hrs intervals).

How often do your sites get indexed in the different SE's?
I, for one, would find it very useful to have my updated content indexed on a daily basis, and would be willing to pay for this if they can guarantee this on major SE's.
But all this is still in theory, as long as they don't have any agreements with the SE's.

That's at least how I read it.
And if that's how it's supposed to work, it still seems to me as a good idea.
Thanks for your input chiyo!