Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: open

Message Too Old, No Replies

How do large websites design their database structure

when there are millions of records

     
8:18 pm on Mar 22, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 3, 2003
posts:960
votes: 0


I wonder how do large websites like Flickr, Photobucket, etc which have millions of photos maintain their database structure. Do they have one single tables containing the records of photos or do they break it up at a certain size.

Does anyone have any opinions to share or could you provide me with some great technical documentation on such complex database structure designing.

8:59 pm on Mar 22, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 30, 2003
posts:374
votes: 0


Well, they might not even store their photos in the database. They could have records pointing to the location of the photos on the file server.
9:52 pm on Mar 22, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 25, 2004
posts:2156
votes: 0


Large binary objects are better held in a filesystem IMHO, since that's what filesystems are optimised to handle in many ways. My experience of filesystem vs RDBMS BLOBs is that the former was 1000x faster for me, and simpler to implement.

Keep your directories small however, so introduce a sensible hierarchy (ie tree structure).

Rgds

Damon

4:23 am on Mar 23, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 3, 2003
posts:960
votes: 0


Well, they might not even store their photos in the database. They could have records pointing to the location of the photos on the file server.

Yes, I am aware of that. They maintain the photos on the filesystem and there are simply records in the mysql tables to reference them. I am however talking about the MySql Table structure. For eg: if there are millions of photos, the records for these millions are held in one single table or they are partitioned [dev.mysql.com]?

9:29 am on Mar 23, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 25, 2004
posts:2156
votes: 0


Speaking only for my mere 20,000+ case: I keep all metadata in memory, and the main table which maps from name to all the other details of a photo/video/sound is flat. With a decent hash of the key there is no reason not to, not even memory these days.

(I generally do not keep photo/thumbnail bulk data in memory.)

Rgds

Damon

1:13 pm on Mar 23, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 21, 2003
posts:1069
votes: 0


The question is not an easy one to answer. I've been working with databases for many years and almost every project I've ever worked on has a different setup.

Do they use partitioning? It's hard to say although it would work. They could very well be using a fulltext text field to hold the tags. They could even just relate the tags table back to the image table. All those would work with millions of rows without any problems.

It all comes down to what the database folks there are comfortable with as long as the results come back as fast as the requirements dictate.

JAG

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members