Forum Moderators: open

Message Too Old, No Replies

! million + entries

         

Sarah Atkinson

2:48 pm on Dec 19, 2005 (gmt 0)

10+ Year Member



My aunt and I were talking aboult the database she works on. They have over 1 million entires could mysql handle that much data? The are curretnly in a contract for 800k to have a company create for them a proprietery database with a webbased application.

txbakers

3:51 pm on Dec 19, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



no problem

arran

4:02 pm on Dec 19, 2005 (gmt 0)

10+ Year Member



In terms of space, MySQL can handle 1 million rows but how many concurrent users will the site have and what percentage of database calls will be write operations?

arran.

Sarah Atkinson

6:19 pm on Dec 19, 2005 (gmt 0)

10+ Year Member



MY guess is somewhwere around 200-500 users for the database and maybe around 2000-5000 write db querrys dally.

dmorison

6:44 pm on Dec 19, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



No problem at all, but you might want to look at using InnoDB tables rather than the default MyISAM. Read up on the way the different table types use locks during SELECT / UPDATE queries as this can have a big impact on the speed of your app for your users.

I have a MySQL application online running at around 50 update operations per second on an InnoDB table of over 4 million rows, with slow queries and inserts going on around that all the time. Works great :)

arran

10:26 pm on Dec 19, 2005 (gmt 0)

10+ Year Member



50 update operations per second

Good job.
Other than row level locking, what steps have you taken to make the system scale up this well?

arran.

dmorison

11:54 pm on Dec 20, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



what steps have you taken to make the system scale up this well?

Embarssingly; almost nothing, other than moving the site onto its own dedicated server as it was bringing my other apps down - but that was a "quick fix", it wasn't until moving servers that I actually sat down and looked at what was going on, and realised that MyISAM's full table locking was the problem.

I will admit however that it is a blindingly simple app. The 50 updates per second is incrementing a counter of type INT(11) based on a single WHERE clause against an indexed VARCHAR(255), e.g.

UPDATE table SET count = count + 1 WHERE this='that';

With that happening 50 times a second load average is hanging around about the 0.2 mark on a 1GHz AMD processor with 512MB RAM.

FalseDawn

9:23 pm on Dec 21, 2005 (gmt 0)

10+ Year Member



Why do you need to update a counter field 50 times per second?

dmorison

10:03 pm on Dec 21, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Why do you need to update a counter field 50 times per second?

It's not one single counter that is being updated at that frequency. The application is recording events, and each event either applies to an existing item (row) - in which case the counter for that row is incremented - or it is a never seen before value in which case a new row is inserted (much less frequent).

FalseDawn

11:54 pm on Dec 21, 2005 (gmt 0)

10+ Year Member



I realized it wasn't just the value in one row (from your query you gave), I was just wondering why you need to do this at that rate at all? What is the purpose of these queries?

dmorison

5:14 am on Dec 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The application is recording website traffic, and lots of it - although not all mine unfortunately!

FalseDawn

3:08 pm on Dec 22, 2005 (gmt 0)

10+ Year Member



OK - I'm sure you have a reason for doing this rather than using a log analysing tool - just seems like a lot of queries to me!