Having a problem with Memcache clearing variables too quickly via PHP

Memcached Problems

[zhanng]

This is a repost from the Content Management subgroup. Not sure where to post memcached related queries, so I am trying the technologies I use one by one in the hopes that someone will have a solution or maybe some advice.

My knowledge of Memcached is reasonably limited outside of using it via PHP, so I was hoping someone had some insight.

My setup is Centos/Apache/Mysql/PHP. I have installed memcached and am using the following line to start it in my /etc/init.d script:

memcached -d -p 11211 -u root -c 4096 -m 2048

The problem is quite simple. Regardless of what I use as a timeout for any of my pages, it works for anywhere between 1 minute and 5 minutes then hits the DB again. The majority of my pages are cached for 24 hours, while others for 30 minutes.

My code for caching is very simple, and I cache everything in seconds using code similar to the following for my parameter: 60*60*24.

Does anyone have any idea what I should look at? Maybe I should dump Memcached for something else? I am open to any suggestions, although I would prefer to stick with memcached simply due to the amount of work it will take to switch it out. Is there a way to monitor Memcached to see what is currently being stored and why things are being cleared?

[eelixduppy]

Please do not repeat your posts in multiple forums.

>> it works for anywhere between 1 minute and 5 minutes then hits the DB again.

I've never used memcached before, but is it possible that the cache is filling up too quickly and the least recently used keys are being purged from memory? If this were the case, it would need to reach out to the db before the configured expiration time if that key were needed again. Perhaps try increasing your cache memory to see if it changes the behavior.


>> Is there a way to monitor Memcached to see what is currently being stored and why things are being cleared?

Run memcached in a screen session with -vv or -vvv to have it print what it's doing. You may also use MaatKit to use tcpdump to analyze traffic to an instance.

Additionally, you can call the statistics commands via telnet: [code.google.com...] or check these tools: [code.google.com...]

[zhanng]

Sorry about the multiple forum posts, I will avoid that in the future.

As for Memcached filling up, that was my original thought as well. However, I have set 2 gigs of space asside for memcached (at least, thats my understanding given the command used to launch it) and my entire site is less than a quarter that. I can't imagine that Memcached is running out of space.

However, another possibility I have been entertaining is that my server was running low on memory so it was forcing Memcached to use less. Is this a possibility? Or, does memcached use swap space when memory is low (what I would suspect)?

I will review those links you sent shortly and try monitoring it more closely. Hopefully it will shed some more light on this.

[eelixduppy]

It does in fact swap pages unless the -k option is enabled.

Also, found this:

-M
Disable automatic removal of items from the cache when out of memory. Additions will not be possible until adequate space is freed up.

This would confirm if the issue is the cache filling up before expiration, but you should also see this by monitoring it.

Hope you can get more insight into the problem. Sorry I cannot be of more help.

[zhanng]

I greatly appreciate your help. I decided to try out phpMemcachedAdmin in order to monitor my memcache. The results are a bit confusing, but telling. For starters, it looks like there is a ton stored:

Current Items: 13716
Total Items: 169980

However, memory usage is practically zero:

Used: 40.7 MBytes
Total: 2.0 GBytes
Wasted: 29.9 MBytes

Something seems wrong here. If my cached data didn't work for a few minutes I would suspect that the PHP code was wrong. I am beginning to think that there is some conflict somewhere, but have no clue where to look.

Would you, or anyone else, know of a good memcached resource? I am a relatively decent developer, but know system configuration on a moderately novice scale.

[zhanng]

I found my problem, and as usual it was my fault. There was a piece of code that would flush the cache every so often. When I removed that code, everything worked perfectly.

Thanks for the tip on phpMemcachedAdmin ... I would have never found the problem without it. I noticed that the Flush count kept increasing. By doing a search, I quickly found the poorly written code.

[eelixduppy]

Great, glad you were able to get it resolved. :)