Forum Moderators: open

Message Too Old, No Replies

Database Synchronization?

multifaceted

         

Salain

11:26 pm on Dec 16, 2005 (gmt 0)

10+ Year Member



I'm developing a set of parsers to collect widget information off of the web and load it into a mysql database. Because of the nature of the widget information that I am gathering, it is necessary for it to be very up to date(within a minute, likely less). Also, because some of the websites would object to the rate at which I'm collecting information, I will need to be collecting the info on different servers with different providers simultaneously.

My current architecture plan is this - I will have a set of computers collecting information, one of these will host a mysql database that all of the others will log into to post updates. That part is all pretty easily accomplished

Now here's the part I have some questions about. My webhost (yahoo) blocks mysql from direct remote access. This, along with the fact that the data collecting programs do a fair number of sql SELECTS that will cause a lot of stress on the main database and scaling considerations have led me to look at synchronizing the main database to the one the website uses.

I gave all of this background because there are a lot of specific considerations...1) foremost I need the synchronization to be FAST, practically realtime. 2) I need it be able to utilize http tunneling to connect to my restricted access webserver mysql DB. 3) The webserver is going have reasonably high levels of traffic, so I don't want all of the remote programs adding to its load. 4)I really might be missing an easier solution, so feel free to point one out if I am.

Currently I'm looking at SQLyog and mySQLs replication feature. Any hints on how people with more experience would set this up are greatly appreciated, I have not used SQLyog or the replication feature before. My current inclination is to try to get mySQL replication running with SQLyog's HTTP tunneling.

Thanks for any thoughts on the subject