Upper limit of files in a single directory

[Nutter]

Is there an upper limit to how many files should be in any one folder before performance starts to degrade? I've got a application I'm writing set to create a new folder to store files after 500 files, which can then be split into 6 different files for a total of 3,000 at the most. My concern is that it's possible, although unlikely, for an installation to create thousands of folders.

[wheel]

I bought a website with 32K pages; the fellow didn't want to sell me the script. Instead he sent me 32,000 flat html files. They all sit in one directory and there's absolutely no performance issues.

[MattyMoose]

You may run into problems when you try to grep or find, or rm.

I've encountered situations where I typed: "rm *", and it repsonded with "rm: too many arguments", because there were too many files. This is FreeBSD, mind you, so linux may be different.

Matt

[Nutter]

It's a PHP application, so I'm not too worried about having to go in to the command prompt to mess with files. The script does have the ability to delete files, but it's done one file at a time so hopefully your 'rm' example doesn't apply. But I'm also only looking about 3,000 at the most, so I'm not getting close to the 32k mentioned above.

[StupidScript]

Where what we call "files" are stored and how many "files" there are will have no impact on performance regarding the requesting and display of a single (or few) files at a time.

1) Hard Drive issues
Many thousands of files stored in many hundreds of locations and all being called by the same request will cause the hdd to work harder (platters spin more, reverse direction more and the heads will shuttle more frequently as they move around the platters). This would be a tiny performance hit due to the physics of the hard drive activity during seek-and-compile.

2) Filesystem issues
None, really, unless you're running Windows, which would need to be "defragmented" on occasion to minimize disk travelling. Cross-partition/drive requests may take a hit as mentioned in the hdd note above.

"Files" are not really "in" one place. They are clusters of data that share some common identifiers in the data packet's headers that indicate (a) which other packets are associated with each, (b) in what order they are to be re-assembled for use and (c) which "folder" (a naming convention, really) "holds" the "file", among other things.

You could have a system with one giant directory that holds every file you fill your hard drive with and there would be no performance hit ... millions or billions of files, if your drive was large enough and the file data small enough. Of course, there are many security and operational issues that make that scenario less than optimal, but which "folder" the "files" are in is not one of them.

Go nuts! :)

[StupidScript]

It occurs to me (for completeness) that a Windows system does have some sensitive 'directories': Desktop, Recycle Bin and My Documents are all rather 'special' in that they can only 'contain' around 400MB of data before it starts to radically affect system performance.

[eeek]

Is there an upper limit to how many files should be in any one folder before performance starts to degrade?

For what file system? It'll be vastly different for ext3 and reiser.