Forum Moderators: open

Message Too Old, No Replies

Large arrays in JavaScript

         

DrDoc

9:07 am on Mar 7, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



A site I'm working on requires the use of quite large arrays in JavaScript.

The array has about 10,000 elements. Each element has three sets of data. I want to use the array to 1) create a number of identical dropdowns with one option for each element 2) retreive the data for each element when that option is selected.

As far as I can see it, I have the following options.

a) Create an Object() with 10,000 elements. Each element is an array with three elements. Use for(## in ##) to document.write() <options>.
b) Create two arrays. One holds the 10,000 "IDs" (used to dynamically document.write() <options>). The other is an associative array with data. Each element in the array being an array itself.
c) Create an Object() with 10,000 elements. Each element will hold the data separated by commas. Use for(## in ##) to document.write() <options>. Use split() when using the data.
d) Create two arrays. One holds the 10,000 "IDs" (used to dynamically document.write() <options>). The other holds the data separated by commas. Use split() when using the data.

What would be the most efficient way of handling this? Speed is the most important factor.

And no, I can't use a server side solution. I would if I could

isitreal

5:24 pm on Mar 7, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If this wasn't drdoc asking I'd question the sanity of the person posting this, but since it is I got curious...

I tried a sample run through with this:
<script>
id_array = new Array();
open_select = '<select>';
close_select = '</select>';
open_option = '<option id="';
close_option = '</option>';
li_items = open_select;

for ( i=0; i< 10000; i++)
{
id_array[i] = '_' + i;
li_items += open_option + id_array[i] + '">' + id_array[i] + close_option;
}
li_items += close_select;
document.write(li_items);
</script>

That's about 1/4 of the functionality you described, maybe 1/6, and that took about 17 seconds to just create the page, and that's with only a single document.write, on a fast computer, if you had multiple d.writes that would obviously increase the time dramatically. And that's not including the much larger data arrays you're describing, it looks like you'd be somewhere around 1 minute to generate the page on a very fast computer, several on a slower one.

It's not clear to me what you are actually setting out to do, if you don't have to write out all the options, it's radically faster of course. If you can't use server side scripting then where is the data array data coming from, you're not typing it in I assume?

figment88

6:34 pm on Mar 7, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



you don't just have to worry about speed but also crashing people's computers.

You have no idea what type of computing resources are available to each surfer. Offloading all of the computation to their browsers may put them over the edge.

What does it mean that you "can't use a server side solution?" Are you just referring just to server side scripting (PHP, ASP, etc.), or cannot you also not use a multi-page (or iFrame) selection process.

DrDoc

9:58 pm on Mar 7, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



server side solution

The data is extracted from a MySQL database using PHP. But to actually write all that data to the browser (generating the dropdowns server side) will make the page a couple megs :(
Even with just sending the arrays to the browser makes up about 200Kb.

To give you some more insight into what it's doing.... It's a quickorder form. Just under 10,000 items. For each item I have the item ID, price, short description, and available quantity.

The item IDs are displayed in the dropdowns. When you select an item it outputs the description, price, qty next to it. They form has five lines with the ability of inserting more lines (using JavaScript).

isitreal

10:16 pm on Mar 7, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



That really seems like it should be done server side, doesn't seem to be a problem, just use an iframe for the description stuff, have that page generated by having javascript load the iframe content page, which in turn would use some kind of passed parameter, or cookie, to write out the correct information server side, that would save about 2/3 at least of the data from having to be on the page, the computer might crash if you ask it for that much processing, as the other person noted.

But if all 10,000 select/option items need to be physically on the page on first load, even just using javascript, you're still dealing with a huge amount of page rendering time no matter what, that would be completely unacceptable as far as I can see, I draw the line at around 15 seconds, over dialup, that would go well over that.

I've heard of flash being used for this kind of thing, it can do live interaction with the backend programming, but I've never done it, but it sort of sounds like something that might do it for you in this case, unless you're like me and never use flash.

figment88

10:54 pm on Mar 7, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If the item ID's are displayed in the dropdowns, does that mean people know the ID?

If so, why don't they just enter it in a form text field and hit enter?

Purple Martin

11:19 pm on Mar 7, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



figment88 is right. It's time to think about useable ways of finding the desired product. Even if you do have to display all 10,000 in one page, can you also offer an alternative page with another way of searching for the item?

For example searching for ID, or for part of the description.

It might also be a good idea to offer people the choice of selecting a desired price-range to cut down the number of displayed items.

My bank's credit card reward program offers all these choices for selecting which product to claim against points.

DrDoc

1:59 am on Mar 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, they know the item ID. I could leave out the dropdown entirely... But I can't wait till page 2 to display the decription and stuff. Description, price, and qty on hand must be displayed on the same page.
I can't use an iframe though... And I can't split the 10,000 items up.

Basically, look at it this way: it's a quickorder form where they enter all the item IDs for what they want to order, then add it all at once to the cart. They are not interested in browsing, they just want to enter the item ID, and the qty, and be done. But, they want to make sure the desired qty is available, and get a brief description to confirm that they entered the correct item ID... and price to calculate how much it will be before submitting the form.

There are other browsing functionalities... but that's separate from the quickorder form.

Purple Martin

2:46 am on Mar 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I'd imagine that overall performance would be better like this:

Give them a page with no list at all. It has a form for the user to enter as many IDs as they want.

The form gets submitted.

A confirmation page is displayed with only the required IDs with full descriptions and prices.

The user clicks a button on the confirmation page to place the order.

The user should also be able to cancel from the confirmation page, or delete/edit one or more product IDs and then reconfirm.

Does all that make sense?

DrDoc

6:25 am on Mar 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, it makes sense... But it's kinda missing the whole point.
They can already do that - elsewhere.

There is a whole order system with different ways of order, searches, confirmations, etc, etc. But this is a sheet that allows them to real quickly place an order for the few items they have. They can currently do what you're describing -- enter any number of IDs, submit, get a confirmation screen.
But these guys place tons of orders. And sometimes they mistype an ID or so. Or, they don't get notified about available quantity till the next page. Some of them are like "ok, I have planned on spending $2000 on the order this week... let's see what that will get me", which is why all that has to be on the first page where the ID entry takes place.

If anything I can get rid of the dropdowns. But I still need to load all the other data and parse through it.

Still... getting rid of the dropdowns saves an immense amount of JavaScript rendering. It also makes the search much faster. It's 300Kb less for the JavaScript to render.

So, that puts me at:

a) Create one Array(). Each element in the Array() is an Array() itself (holding price, description, quantity).
b) Create one Array(). Each element in the Array() is a comma-separated list of data. I'll have to use split() to extract the different pieces of data (which isn't necessarily a big deal, since I always use all three together).

So which one of the two do you think will perform best? Which one would you use, and why?

john_k

6:24 pm on Mar 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I just finished a site that used some large javascript arrays. One recommendation I have is to offload as much as possible into an external javascript file. It can still be generated dynamically, but it can be made to look static to the browser. A good candidate for this would be your descriptions. The prices might work well this way also. You can add a parameter to the javascript file's URL to ensure the file is periodically re-generated. A session id works good for this.

With that method you would use parallel arrays, putting your available quantity in one array that is always loaded with the page and the descriptions in a second array located in the external javascript.

The server would track whether a given session has retrieved the description array yet or not. Once it has, then future requests for the array from the same session would just return an HTTP 304 status.

The primary benefit of doing it this way is that if they visit the page more than once, subsequent page loads will go faster.

Also, if you do utilize ANY external javascript files, then the other recommendation I have is to detect whether or not the external javascript is loaded or not. You can do this by setting a variable to indicate whether or not the external file has finished loading. In the head of your page, initialize the var to false and add a function that can be called to set the variable to true. Add a call to this function at the bottom of your external javascript file. Then, on your functions that try to display descriptions and prices for entered IDs, check to make sure all is loaded. This is all to prevent javascript errors that will inevitably occur because the user starts doing their work before everything is loaded.

john_k

6:30 pm on Mar 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



One other tip I forgot to mention: set the default mouse/cursor style to an hourglass. Then add javascript at the end of the page (or the function called after all is loaded) to set the cursor back to the default. This will also help to cut down on the clicks and data entry that occur before everything is loaded.

DrDoc

6:32 pm on Mar 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Now that is some great piece of advise, john_k! :)
I hadn't even thought about that.

twist

7:08 pm on Mar 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



And no, I can't use a server side solution. I would if I could

Do you have the option of using something besides JS? If it was allowed you could write a program in C or VB (if your short on time). The users could download a compiled program that just pulls the data from the database.

DrDoc

8:58 pm on Mar 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Unfortunately, it's JS or nothing :(

Purple Martin

10:20 pm on Mar 8, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



So, that puts me at:

a) Create one Array(). Each element in the Array() is an Array() itself (holding price, description, quantity).
b) Create one Array(). Each element in the Array() is a comma-separated list of data. I'll have to use split() to extract the different pieces of data (which isn't necessarily a big deal, since I always use all three together).

It would take a guru like you only a couple of minutes to build two test pages, one for each method (you don't have to worry about making the output pretty at this stage). Time them both, and see which finishes fastest! That's the only way to get a genuine answer to your question.

I also like the idea of using .js files... it sounds to me like your customers are regulars, so having the arrays external means the customers don't have to download the 200kb every time. I suggest you split the data across, say, 10 .js files of 20kb each, so that if one product changes they only need to re-download 1/10th of the data.

Finally, as well as using the hourglass cursor, give them some text: "Building product list, please wait a moment".

AWildman

5:57 pm on Mar 9, 2004 (gmt 0)

10+ Year Member



Can you do an array of objects that hold the pieces of info that you want? It would be much simpler to output

var prod = new product_info('timmy', 2, 5, $1000);

function product_info(name, number, quantity, price)
{
this.name = name;
this.number = number;
this.quantity = quantity;
this.price = price;
}

document.write(prod[i].name);
document.write(prod[i].number);
.
.
.

qeantk

9:28 pm on Mar 21, 2004 (gmt 0)

10+ Year Member



I really don't see forcing your users to deal with (download, render, and UI-wise even) a full 10,00 item list as that much more friendly. Generally, if you run into that severe of problems with what the client-machine can handle, the general precept and UI itself have some problems.

That said, I think there is a work-around that is the best opf both worlds. This is a perfect case for a RPC. Have a text box for entering the id and on either onblur or onchange, with a check for the length of a valid id, call a function that dynamically creates a <script> element with the id in the querystring and appendChilds it to the <head>.

Then have your server-side stuff just output a function into that .js that updates the description, price etc. This can actually be pretty snappy, and this is a case where I thik the performance of that small call to the server will be MUCH more responsive on the client-side.

DrDoc

10:52 pm on Mar 21, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> not user friendly

Well, it was requested by the user...

> not having the full list

They want to be able to navigate up and down in the list... A call to the server is slower than getting the data from a JavaScript array.

qeantk

11:04 pm on Mar 21, 2004 (gmt 0)

10+ Year Member



Obviously, client requirements trump, but I have just seen plenty of times where they ask for something, but once they see how it works out in Real Life, they realize they didn't want what they thought.

I thought you said they know the id, and can just type it in, but you just didn't want them to wait for a full page load to see the results. This would still require a round-trip, but it's as small of a round-trip as you can make, and no page refresh. Sometimes I have really found that sort of thing to perfom much better than dealing with passing silly large datasets around, and trying to crunch them in .js, and dealing with the intial slow download.

Of coure, if (you would know best) it really needs to be all passed to the client (such as your syaing they want to be able to go up and down the list,via arrows or whatever), then the already made suggestions about caching as much as you can in different files is your best option. I am willing to bet actual time on these roundtrips is still going to take longer than just doing a small RPC (barring the whole manual list navigation thing), but I guess percieved time is all that matters, and that is one you'll always have to just test for.

Good luck, either way.

Wait, so are you doing a text-box that sets the current select on the list, then, or are they supposed to pull-down a 10,00 item select and find the one they want (as IE does a crappy "find in select" natively)?