maximillianos - 7:45 am on Jun 2, 2010 (gmt 0)

Sorry for the late reply. Project has been on the back burner. I did end up writing the first program, to save the state of my memcache.

I ended up choosing to save it to a separate database instead of a flat file. This gave me much more control over the data, and doesn't tie up my live database while re-loading the data.

My test runs loaded over 75,000 pages of full cached html pages into a separate database (drop table and then inserts) in under 2 minutes. I also kept the state of the dirty bit as well, so when I restore it, the site/software will know the state of each page in memory as well. The dirty bit is a custom memory field I created to flag when source data has changed that affects a page.

I've also written the memcache_reload.pl script. It simply loops through the records and restores all the data to memory. This piece I have not tested yet, since it involves essentially bringing my server down and back up. I'm planning on doing some heavy testing during a late weekend night when traffic is at a low point.

This solution has also prompted me to think of other utilities I want to build. For instance sometimes I need to make a small change to a page template, however doing so involves reloading 75,000 generated pages from the database. I'm designing a program that will simply take a snippet of code and do a replace on it with a new snippet in the memcache memory. This will allow me to make minor tweaks to the template and not have to start all over re-building my memcache from scratch. The process would backup the memory as I do on a reboot before making any changes, just in case a problem occurred. I could then restore the state of the memory back to what it was if needed (using my first set of programs...=).

Yeah, it is late here and I'm probably biting off more than I can chew, but it is definitely a problem I need to solve since my hands are typically tied when it comes to making changes to these heavily cached pages. I can't afford to have my server bogged down for half the day whenever I need to tweak one of the page templates.

I'll stop back when I have gotten a bit further. I'm confident the first process is going to work fine (saving and reloading the cache). Not so sure about my second idea of replacing snippets of code directly in memory. =)