higher speed or better quality - which to choose

if you cannot combine both?

[idolw]

we run an affiliate website with many widgets from various merchants.
the visitor uses our own search to see items available on stock. when users make a search our website returns a list of results sorted by price (from cheapest to most expensive) and allows several possibilities to sort and narrow the results to allow the user find exactly what they want.
our site communicates with merchants' systems via XML. in order to be able to provide the user with our search features we need to query the merchant's server about availability and price of each item we list.
one of the merchants we work with returns the results extremely slowly (up to 50 seconds to display the results on our webpage). we hate it and only use this merchant to sell the items where they are the best. i have already discussed the possibility of speeding up the work with their technical support team and the only solution is as follows:
instead of returning us information about each item we ask for, they can return us a certain number (say 25-35) of items that are found by their system first.
this would speed up the search from 50 seconds to 7 seconds but:
- we would not be able to allow the user to sort and narrow the results the way they can now
- combining this merchant's results with other merchants' results becomes impossible.

one important fact is that the user has to wait only once - when doing the very first search. everything is kept in a session later.

so my problem is: speed or quality?
shall we give less and do not ask for waiting?
or give much more than 97% of websites of our type and ask the customer for patience?

thanks

[harikrishna]

As far as I can tell from your post, you are searching several merchants remote datafeeds dynamicly every time someone makes a search? If I have understood you right that is a very bizarre solution.

Try a database.

[idolw]

yes, searching several merchants at the same time everytime someone makes a search.

i have a database of items (and their specifications) but the availability and prices change every minute so must always check it to provide real data.
the problem is the connection time between my server and the merchant's one.

[harikrishna]

then your server should fetch the feed every minuite and update your database. that way your users wont have long waits. it should be more efficient for your server too.

[idolw]

it is impossible in my sector. Too many searches would have to be performed.
My site is a hotel website.

[harikrishna]

Your market is irrelevant. As my understanding is, your currently searching the merchants datafeeds for their room prices or whatever.

This is always going to be slow, it makes much more sense for you to setup a cron to fetch the datafeeds and load them into a local database every minite. Then when the user searches you query your local database for the results.

I cant see why my solution would not work.

[Morgenhund]

Fetching data every minute seems odd anyway!

You should use database solution, but update your data sparsely, say 1 time per hour/day, and just have a disclaimer that you are trying to provide as accurate data as it goes, but without warranty.

Trust me: you can update your data every second, but there _always_ be an outdated feed -- say just because a hotel has not updated their data timely, or just two reservations come along, or whatever.

It's a real world!

[LifeinAsia]

Your market is irrelevant.

Not at all! Hotel inventory & even prices can change by the minute. Much different from a widgets site that changes their prices once a year and never even comes close to running our of inventory.

This is always going to be slow, it makes much more sense for you to setup a cron to fetch the datafeeds and load them into a local database every minite.

Think about what you're asking! Even if we're just talking about U.S. hotels, there are over 45,000 GDS-enabled hotels (double that if you're doing world-wide reservations). Even figuring that each hotel has a basic inventory of only 10 rooms/night (many have hundreds), that's still a lot of data. How far out are you going to search? One month? 3 months? 6 months?

Do the math- this is a HUGE amount of data! Even if an affiliate had their own database that could deal with this, the cost from all the traffic would wipe out profits. And the parent company would never stand for thousands of affiliates dumping the entire database every minute- that would kill THEIR profits!

I cant see why my solution would not work.

Can you see it now?

[edited by: jatar_k at 5:20 pm (utc) on Jan. 31, 2006]
[edit reason] fixed quote [/edit]

[grandmf]

Yes, you do have to do real time lookups. Ways to improve would be to have the user put in as much data before hand to keep the number of records down. Instead of seaching all hotels in FL, make them pick the sorting before they click. Make them pick a city, or a small group of cities, size of the room, cost range, pools, etc. Then you can send your XML feeds out to your vendors and get a smaller subset thus cutting down time.

[idolw]

how can i check it every minute?
say: one hotel, 6-month perspective, average stay lenght between 1,2 and 3 nights.
I would have to check availability and pricing for every single day for each stay lenght. so only if we use full numbers only, in 6-month perspective, it would be 180*3=540 queries about just one hotel (or even room type in a hotel) every minute. and yes! i have to check for 1 day, 2 days and 3 days as the prices annd availability differ.
so what about a place where i have 100 or 350 hotels? and there are plenty of such.

by the way: can anyone sticky me with an URL do a website that is doing this? It would be interesting to see something I do not believe exists.

[andye]

Maybe you could return the page to the user quickly but with gaps, then use AJAX to 'fill in' the missing details as they arrive?

That way the user doesn't give up before the page loads.

hth, a.

[idolw]

grandmf,

yes, that is what we are trying to force the user to do. the problem is that whoever comes, wants to see everything that is available first. also, when you ask them to specify their neeeds in a very detailed way, the number of results they get may be discouraging :(

that is why i asked what would you put your money on: speed or features?

[LifeinAsia]

Maybe you could return the page to the user quickly but with gaps, then use AJAX to 'fill in' the missing details as they arrive?

Take a look at <snip> they get feeds from multiple sources and dynamically re-sort as more data comes in. Still rather slow for the full results, but the user can stop the requests at any time and go with what's already been returned.

[edited by: lorax at 6:47 pm (utc) on Jan. 31, 2006]
[edit reason] removed DN [/edit]