homepage Welcome to WebmasterWorld Guest from 54.161.175.231
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
Forum Library, Charter, Moderators: bakedjake

Linux, Unix, and *nix like Operating Systems Forum

    
Upper limit of files in a single directory
Nutter




msg:909921
 4:20 pm on May 14, 2006 (gmt 0)

Is there an upper limit to how many files should be in any one folder before performance starts to degrade? I've got a application I'm writing set to create a new folder to store files after 500 files, which can then be split into 6 different files for a total of 3,000 at the most. My concern is that it's possible, although unlikely, for an installation to create thousands of folders.

 

wheel




msg:909922
 2:46 am on May 15, 2006 (gmt 0)

I bought a website with 32K pages; the fellow didn't want to sell me the script. Instead he sent me 32,000 flat html files. They all sit in one directory and there's absolutely no performance issues.

MattyMoose




msg:909923
 10:18 pm on May 16, 2006 (gmt 0)

You may run into problems when you try to grep or find, or rm.

I've encountered situations where I typed: "rm *", and it repsonded with "rm: too many arguments", because there were too many files. This is FreeBSD, mind you, so linux may be different.

Matt

Nutter




msg:909924
 12:56 pm on May 17, 2006 (gmt 0)

It's a PHP application, so I'm not too worried about having to go in to the command prompt to mess with files. The script does have the ability to delete files, but it's done one file at a time so hopefully your 'rm' example doesn't apply. But I'm also only looking about 3,000 at the most, so I'm not getting close to the 32k mentioned above.

StupidScript




msg:909925
 10:11 pm on May 24, 2006 (gmt 0)

Where what we call "files" are stored and how many "files" there are will have no impact on performance regarding the requesting and display of a single (or few) files at a time.

1) Hard Drive issues
Many thousands of files stored in many hundreds of locations and all being called by the same request will cause the hdd to work harder (platters spin more, reverse direction more and the heads will shuttle more frequently as they move around the platters). This would be a tiny performance hit due to the physics of the hard drive activity during seek-and-compile.

2) Filesystem issues
None, really, unless you're running Windows, which would need to be "defragmented" on occasion to minimize disk travelling. Cross-partition/drive requests may take a hit as mentioned in the hdd note above.

"Files" are not really "in" one place. They are clusters of data that share some common identifiers in the data packet's headers that indicate (a) which other packets are associated with each, (b) in what order they are to be re-assembled for use and (c) which "folder" (a naming convention, really) "holds" the "file", among other things.

You could have a system with one giant directory that holds every file you fill your hard drive with and there would be no performance hit ... millions or billions of files, if your drive was large enough and the file data small enough. Of course, there are many security and operational issues that make that scenario less than optimal, but which "folder" the "files" are in is not one of them.

Go nuts! :)

StupidScript




msg:909926
 9:04 pm on May 31, 2006 (gmt 0)

It occurs to me (for completeness) that a Windows system does have some sensitive 'directories': Desktop, Recycle Bin and My Documents are all rather 'special' in that they can only 'contain' around 400MB of data before it starts to radically affect system performance.

eeek




msg:909927
 6:06 am on Jun 2, 2006 (gmt 0)

Is there an upper limit to how many files should be in any one folder before performance starts to degrade?

For what file system? It'll be vastly different for ext3 and reiser.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved