sitz - 1:30 am on Apr 30, 2005 (gmt 0)

can i take the drive which currently has the inappropriate filesystem and remount it with the appropriate filesystem?

No, the partition will need to be reformatted. This will (obviously) destroy the data on the disk, so you'll need a migration path (this may be as simple as scheduling downtime where you back up your data, reformat the disk, and restore the data). I'd strongly suggest experimenting with filesystems on test system, though! =).

Also note that support for the filesystem you're using will need to be present in your kernel if you want to make use of it.

As to the speed, it should open a file in a directory of 1,000,000 files as fast as it would in a directory with 10 files; just don't do any kind of file globbing operations, or things will get REALLY ugly REALLY fast (ask me how I know). You could clean up things a *bit* by making your directory structure deeper:

..../e/a/s/eastwood.suffix

...but really, a SQL solution is likely the right way to go. I don't claim to be a DB guru, but I can tell you a few things:

* the structure of the database is critical
* proper uses of indexes is critical
* table joins are not cheap; they're very useful, but if you don't have to use them for a particular operation...don't.
* integer comparisons are cheaper than string comparisons

So, for instance, let's say that you have a form with a drop-down with a list of $somethings. You'll then take the submitted value and do something like:

SELECT * FROM <Table> WHERE <column> = '<submitted_value>'


When you populate the drop-down, populate it from a database table which maps integers (ids) to strings. Then set up the drop-down like this (populated from the database table):

<SELECT name="stuff_on_my_floor">
<OPTION value="1">Carpet
<OPTION value="2">Sofa
<OPTION value="3">Dustbunnies
</SELECT>


Now, when the data gets submitted, you can do:

SELECT * FROM <Table> WHERE item_id = '<submitted_integer>'

...instead of something like:

SELECT * FROM <Table> WHERE item_name = '<submitted_string>'


Queries can also be optimized; one of the DB folks at my place of employment optimized a query from tens of seconds down to a few milliseconds. Other things to note:

* perl/python/whatever CGIs do not scale; languages with embedded interpreters (mod_perl, PHP, etc) scale far, far better

* use database connection pooling; setting up (and tearing down) connections to a database is a waste of resources. Note that database connection pooling can NOT be done with fork()'d CGIs, since they exit between invocations.

(edit: spelling correction)

[edited by: sitz at 2:04 am (utc) on April 30, 2005]