Too many folders on image database site?

architecture as data structure

[millercia]

My in-progress Virtuecart ecommerce site will have around 4K different images of about 1500 unique items to star with, with from one to eight "views" of each item. More ongoing. Thumbnails and medium versions of the images will all be resized in HTML from a large (775px X 1000px) JPEG. I have renamed all the files with keywords in hopes of getting a boost in Google Images.

The subject items were each created by (different) unique artists and images are now in about 275 folders, one for each artist. This how the current version of the site (works well) is organized. The programmer has proposed placing the main view (1) and secondary views (from 0 to 7 additional) for each item in a folder for each item, with a vague rationale that it will help with site searches. I can only imagine that the intention is to use the "physical" organization of the site (directory structure and filenames) in place of or to simplify URL rewrites. I have no problem with simplicity but I see 1500 plus folders as a red flag. From an organizational viewpoint this seems like increasing entropy, as the "artist" field is one of the few things otherwise similar items don't share. Without a native human-readable structure maintenance might also be a nightmare. I also wonder if 1500+ subdirectories might create unanticipated performance issues. I can see this approach working well on a site with far fewer items but I can't shake a feeling of dread, perhaps mistaken but based on experience in maintaining the old site.

Would appreciate opinions. Might the tail indeed be wagging the dog here, or am I seeing the red flag through fear-googles?

[phranque]

this is a file system issue rather than a web server issue.

as far as the file system is concerned, everything is a subdirectory of the root directory, so 1500+ subdirectories is trivial.

[millercia]

this is a file system issue rather than a web server issue.

as far as the file system is concerned, everything is a subdirectory of the root directory, so 1500+ subdirectories is trivial.

Thanks for reply.

Am I correct in thinking that images would, per standard practice, go into (probably subdirectories of ) the Apache images directory? The intention is to place all of these 1500 folders into one product images folder. Could too many items in one directory cause any performance issues and if so, what's a safe limit?

[lucy24]

Thumbnails and medium versions of the images will all be resized in HTML from a large (775px X 1000px) JPEG.

:: detour to put on 'user' hat ::

Nooooooo! Keeping only one version of each file in your database is fine. But any resizing needs to be done server-side. Otherwise the site will be seen as inexplicably slow-loading. This is not only a human-interface issue; search engines measure page access time too. Your server can resize an image in much less time than it takes a human to download the oversized version.

[Kendo]

any resizing needs to be done server-side

The down side of this is that it places an unnecessary load on the server. Take for example a page of thumbnails resized by the server from larger images. Server resources are consumed for every image resize which needs to be performed as the page loads.

So if you have 1,000 concurrent users loading one thumbnail page you can have resource issues and how many more thumbnail pages are there to spider and explore? I used to use resize server side and used that method for several sites including client sites (home grown shopping cart) that included shopping carts and listings for 1,000s of items and their images.

Now I leave image resizing to the end user. Better to use their computer resources rather than to stall a server.

[lucy24]

it places an unnecessary load on the server.

(Emphasis mine.)

Better to use their computer resources rather than to stall a server.

Y'know, when I first started reading these forums, I was struck by the widespread attitude of contempt for the user. Who cares what the customer wants or why they want it? It's my site and I'll do it my way. Or, in this case: Why bother to do your own job if you can unload it on someone else?

[millercia]

Lucy:

But any resizing needs to be done server-side. Otherwise the site will be seen as inexplicably slow-loading. This is not only a human-interface issue; search engines measure page access time too. Your server can resize an image in much less time than it takes a human to download the oversized version.

Kendo:

The down side of this is that it places an unnecessary load on the server. Take for example a page of thumbnails resized by the server from larger images. Server resources are consumed for every image resize which needs to be performed as the page loads.

So if you have 1,000 concurrent users loading one thumbnail page you can have resource issues and how many more thumbnail pages are there to spider and explore?

Thanks for replies. Given the one large JPEG approach, the best way to go seems to depend on traffic load. With client resizing, lots of concurrent users might have a better experience. In a low traffic situation the server would not be overtaxed and so the user would likely benefit from both faster resizing and, since only the displayed size is downloaded, faster loading. Our site is presently low traffic but we certainly hope to increase volume and this was a big motivation for redesigning it.

I'm questioning the wisdom of using a large source image for resizing at all. I don't see how keeping two additional smaller versions significantly increases complexity and the storage increase would be under 10%.

There will be up to eight thumbnails on a page. Thunmbnails will be used to select a medium which when hovered over will bring up a zoom box with the larger image. For some pages, therefore, up to eight 200-250KB JPEGS will need to load before the page is functional.

I had not considered spiders' measurement of load times. This could be more "noticeable" to them since once they connected they would probably initiate a rapid-fire series of page requests.

[topr8]

i have 4 sizes of jpg for each image stored on the server, it works well for me.

why do any kind of dynamic resizing, when space on the server is relatively cheap.

[brotherhood of LAN]

Confusing thread...

It seems the discussion covers
1. # of files in a folder
2. Resizing/optimizing images client-side before upload
3. 1. Creating smaller versions of the same image after uploading on the server side OR 2. just serving the large image and letting the browser resize to the desired size OR 3. resizing the image using server side scripting, on every request.

For 1. It would depend on the filesystem but I would guess anything less than 100K folder/files isn't a big issue.

2. I use plupload and yes it seems to make sense to get clients to use their resources & save network transfer

3. I definitely wouldn't serve a larger image and let the browser resize it. Consider a 1MB image... and 50 concurrent users. You're going to need a 100Mb connection to serve them in a decent timeframe. Resizing them server side on *every* request just doesn't make sense.

IMO the point of having smaller images is that you do it once (even client side if you will), but it will be served more than once, so it makes sense to do it.

[millercia]

Thanks for replies.

topr8:

i have 4 sizes of jpg for each image stored on the server, it works well for me.

why do any kind of dynamic resizing, when space on the server is relatively cheap.

I agree. The rationale was that it would be simpler, which does not make much sense.

brotherhood:

3. I definitely wouldn't serve a larger image and let the browser resize it. Consider a 1MB image... and 50 concurrent users. You're going to need a 100Mb connection to serve them in a decent timeframe. Resizing them server side on *every* request just doesn't make sense.

IMO the point of having smaller images is that you do it once (even client side if you will), but it will be served more than once, so it makes sense to do it.

I see your point. Processing the same thing over and over is crazy. Seems like the best solution is to use multiple sizes stored on the server. It is automatically the least demanding of both server and client resources and really has no real down side.

[lucy24]

Confusing thread...

My bad. I zoomed in on that "resized in HTML" line in the initial post. But I don't think there was anything about users uploading files. There I'd agree that you can perfectly well place the burden on them-- or their computers-- to do the work. In fact it's the same principle: the person providing the file should provide it in the best form.

For most people in most situations, maintaining multiple files on a server is probably much less work than dynamically resizing a single file. One possible exception is if you've got a nice header that you want to fit neatly into the user's viewport.

On the purely mechanical question: I suspect your human ability to keep track of things would fall apart long before the server. And if you're going to accidentally delete or duplicate* a directory, let it be one containing 500 files rather than half a million :)


* A computer doesn't absent-mindedly double-click instead of single-clicking. But a human does.

[thecoalman]

I know on my VPS there is a file/folder limit. It's a huge limit but there is a limit.

The down side of this is that it places an unnecessary load on the server. Take for example a page of thumbnails resized by the server from larger images. Server resources are consumed for every image resize which needs to be performed as the page loads.

You don't have to serve them dynamically and I'm not even sure why you would unless it was special case like you're serving a captcha image.

If you can't do it server side use a tool like irfanview to batch resize and upload with FTP.

Since you mention artists one thing to be aware of is EXIF/IPC is typically stripped using the GD library which is standard way to resize. You need to use Imagemagick or additional PHP libraries to keep that information.

[Kendo]

I was struck by the widespread attitude of contempt for the user

It may appear that way at times, but I am certain that most here are like me and do consider the end user because it is their patronage that we need to make our projects a success.

Now, having said that, as webmasters and web designers who are creating mazes of web applications and trying to make web forms and interactives as user-friendly as possible, we are constantly faced with the problem of making them idiot proof.

There is no room for grey. It needs to black or white especially when it comes to security. What percentage of the world population is right at this moment running software and servers trying infiltrate and exploit our web services?