Forum Moderators: phranque
That's it.
I have tried to tell that I should have some more specifications or that it is not viable for 1 person in 1 month.
Might be I am wrong.
What are the estimates (cost or money or person-months) for developing a metasearch engine?
How much would it cost to buy obe?
I searched Internet to get ideas but still confused:
- MS develops its own metasearch engine for some months with unknown resources;
- there are costs from 10 to 50 USD on Internet with quite unclear capabilities (it sems to me just offering the right to use internal engine on someone else's server)
Thanks for your time!
1. You need to have permission to query the other engines. Otherwise, you run the risk of them blocking your server. Also, if you're scraping their HTML pages, and they change the layout, your engine may be rendered useless. Given this, it would probably be in their best interests to have a long-term support contract with you.
2. Querying the engines and retreiving the results isn't too difficult, but combining and organizing the results intelligently would be a bigger task.
Buying an engine may be possible, but it may not meet their needs, and it may be hard to modify.
Personally, I'd say 3 to 4 months is more reasonable (if you have other projects going on), and you definitely need better specifications. An important question to ask is why do they need a meta engine, what purpose are they trying to accomplish, and what kind of product/site will it be integrated into?
For example, if they're just trying to run a search engine so that they can place their own paid ads, that's understandable, and it helps define the project goal.
I made some investigations and ran some codes, projects. And I am quiet nervous about "no more than a month by 1 programmer"
Thanks for sharing with me your opinion/estimates!
- MS develops its own metasearch engine for some months with unknown resources;
Microsoft developed its own fully fledged search engine (not metasearch), the cost of development (ignoring hardware) is certainly well above $1 mln, most likely $10 mln+, but less than $50 mln.
Not sure about performance of metasearching on top of metasearchers - this will be largely outside of your control, but overall task should not take more than 1-2 weeks -- for me that is. How long would it take for you depends on your experience which nobody apart from you can judge well.
This is not comparable
Really I run it already in HTML giving output in different frames with possibility to choose search engines, metasearch engines or directories. Well, it runs on client-side, though it is not an issue to pass it to server side.
It is not clear how to combine in one list disparate formats, rankinings, repetitions
mhhfive:
I couldn't understand. I am not in (I am neither webadmin, nor lawyerin any way in this area).
What do you mean "to stay under the radar"? banned for what?
developing a crawler is quite an involvement and there is Open Source for this:
I think I know pretty well about it....
But you talking about META-searching -- you don't need crawler for this: html scraping and parsing of results does not qualify for a crawler.
So simple. Why I cannot find any ready free open example, sample, description, project, algorithm, manual, tutorial oe whatsoever on metasearch engine while it is not a problem for more complicated crawlers?
What is the problem with metasearch engine? Why are they sold em masse if it is something bannable or simple?
I still couldn't understand the comment on being banned, for what?
I got it done for $250.00 for a guy in Romania at rentacoder. I think it was a fair price, but after I pay I saw the code and it looked like a highschool homework.
I redid the thing using pure OO within a week.
It metasearches 5 engines, but I didn't released it to the public, I am using it internally in admin pages in one of my site.
[edited by: fischermx at 9:12 pm (utc) on Oct. 10, 2005]
Why I cannot find any ready free open example, sample, description, project, algorithm, manual, tutorial oe whatsoever on metasearch engine while it is not a problem for more complicated crawlers?
Its a good question. Perhaps the answers are:
1) its not worth making yet another meta searcher
2) its so simple that if you need instructions how to do it then its probably best not to
3) something else
You seem not to understand that in order to meta search you need permission (often not given unless you pay) of sites that you going to meta search. From technical point of view building meta searcher is so trivial that its boring to even think about doing one.
If I am asked, I'd like to understand it I am going to be paid. If I am asked smth to be banned, I'd like to understand it. Because I continuously have traoubles in getting payments.
THIS IS ALREADY NOT any JOKE for me.
So, how it is that there are sales of metasearch engines but they are being banned?
Do u want to say that the sites with metasearch are paying to other sites for using their search capabilities?
I am not going to search or scrawl anything. I am going to submit queries and receive results from search, metasearch, directories sies and treat their output internally on "my" server. (It can be even done in browser of those who access the site). The metasearch engine will be on my client's site.
I do not quiet catch. Where us the difference: I google, hotpile or mamma for results through a browser or I use instead of a browser just another type of "browser", call it metasearch engine?
Do u want to say that the sites with metasearch are paying to other sites for using their search capabilities?
They either pay to be allowed to do that, or explicitly allowed to do that or just breaking Terms and Conditions by meta-searching sites that don't allow this without explicit permission.
If it suprises you that search engines don't like meta-searchers then you just lack experience in this area - search engines don't like freeriders.
But again, since you are just contracted to do the job it should not concern you as it will be responsibility of whoever will use your meta searching engine to actually make agreements with sites they meta search.
I think I or we should have used quering and crawling instead of "search" in our conversation somewhere earlier.
You can call it anything you want, but this won't matter much to people who will be metasearched or "queried" if you like.
If you pay depends on client's expectations then make sure they know what they are doing because their little project may fail and they could just think its your fault, even though technically you are in the clear.
