How to sync?

[paleolith]

For years, I've maintained my rather inactive web site solely by manual FTP. It's gotten large enough that I'm finally trying to use a synchronization tool. OK, it got that way a long time ago and I'm trying to catch up.

Since I've been using WS-FTP, I fired up its sync tool. What I found was that it thought nothing on the site matched my local files. Turns out this is a combination of things the FTP server on my web host doesn't support. It doesn't support MDTM, so the server file timestamps are the upload time. It doesn't support any message digest such as XMD5 or XCRC, so the client cannot match files using that.

I tested a second web host that I use for another web site, and it has the same limitations.

So how do FTP-based synchronizers work? I've spent a couple of hours searching the web, for nada. Many sync programs exist, but when you try to find out how and whether they get around these problems, they just sort of wave their arms around.

So is this situation normal? Are sync programs able to handle it? If so, how?

It's easy to see that with a message digest plugin on the server, matching is trivial. I can also see why webmasters don't want this overhead.

It's easy to see that, even with a one-minute granularity, one could use timestamps to determine whether files match, albeit with a small chance of failing to catch a mismatch.

And I can see that a more sophisticated program could keep a database of the timestamps on the FTP server: after initially uploading a file, request info on the file, enter into a database the file ID and the timestamps for the local and remote versions. Then on sync, check the local and remote timestamps and see if they match based on the DB record that links them.

The last of course would require the entire web site be re-uploaded to prime the database. I'm happy to bite that bullet if it's the right one. Whether it's the right bullet to bite is what I'm having trouble determining. The docs for WS-FTP are minimal and don't tell me. I've looked at the web sites for a number of other sync tools and none of them tell me.

Edward

[mack]

I feel your pain on this one, its can be very problematic to find a workign solution, because there are only so many tests you can actually do before trying it live, then you really don't want to know things haven't went to plan. I recommend you do a full backup of the live site (as it is) before trying to do anything, save it in a different location to your development files. (the ones you want to sync) then if things do go wrong all you need to do is recover your downloaded backup.

Is your site, mainly dynamic or static. You mentioned the database, perhaps mysql replication is something that might be of help for this? it wont back up your actual files, but will keep an up to date copy of the database. and sync that.

Mack.

[paleolith]

Thanks. It's all static. When I mentioned database, I was speculating on how FTP sync might work. My master copy is the local one, so if I totally screw up the online one, I just wipe it and re-upload. But that takes a while.

I posted a similar question for WS-FTP and have had no response. I think I'll investigate some of the programs which specifically advertise synchronization as their main feature, as opposed to an afterthought as with WS-FTP. Googling <<FTP synchronization>> gets lots of hits.

Edward

[damon_cool]

Maybe check WinSCP, it has a checkbox for each transfer to "Only upload new or changed files". Not exactly what you are looking for, but it might help. Very handy tool.