It seems like once you get a couple thousand files in a directory it takes forever to ls it. I think that it has to stat all the files and that takes forever. I don't think it's even a linear increase since it starts to get really slow pretty quickly. I can't remember if it slows down the time to stat a file given its name or to stat the directory or anything like that, though.
I would suggest a couple of things: test it out first by making a directory with 100,000 or so files in it and see if it works very well. Also, think about making the directory structure hierarchical in some way. You will notice that machines with many users do this with the first two characters of the username for /home. Something like:
Yep, was planning to have directories 0-255, each with subdirs 0-255 (ie a 16 bit number range mapped to the filesystem). Many directories would in fact be empty or maybe only contain a few files, some might contain a few thousand.
> A relational database is probably starting to sound a little more attrcative
Hehe, actually I'm using one now, and it's starting to drag its feet a little. I think I can maybe cut down the search times by going this way instead... but my grand plan is a little hazy at the moment I have to admit.