Memcached monitoring

Tool

There are several memcached monitoring tools available on the market, such as phpMemcachedAdmin, memcache-top, statsproxy, in addition to the memcahced console accessed by telnet. We will use phpMemcachedAdmin as a monitoring tool: it has a simple web interface and it is easy to use, we will also enumerate a few points which should be monitored and troubleshooting.

PhpMemcachedAdmin installation guide

PhpMemcachedAdmin must be downloaded from the website  https://code.google.com/p/phpmemcacheadmin/, it is a simple script in php and as such it should run on an http server that has php support.

The latest version up to the date on which this guide was written is 1.2.2-r262, which can be obtained at the link: http://phpmemcacheadmin.googlecode.com/files/phpMemcachedAdmin-1.2.2-r262.tar.gz

Following the instructions on the site, decompress the package, give permission to execute the scripts and write permission in the server configuration file and the folder in which the reports are generated

in the root folder of your server e.g./var/www/

mkdir phpMemcachedAdmin

cd phpMemcachedAdmin

wget http://phpmemcacheadmin.googlecode.com/files/phpMemcachedAdmin-1.2.2-r262.tar.gz

tar -xvzf phpMemcachedAdmin-1.2.2-r262.tar.gz

chmod +rx *

chmod 0777 Config/Memcache.php

chmod 0777 Temp/

How to configure

Point the browser to install phpMemcahedAdmin on your server, in this case our installation is in http://localhost/phpMemcachedAdmin.

Initially the monitor is already configured to connect to a local memcached on the machine on which it was installed; if the memcached is installed somewhere else, you should access the "Edit Configuration" link located in the upper right corner, which will direct you to the page below.

On this screen you can monitor several memcached installations, for our configuration let's just edit the server address and port, keeping in mind that the default port is 11211; after saving the changes, follow the monitoring pages by clicking "See Live Stats" or "See Stats for" plus the server selection in the combo.

If the settings present a problem, you will view the page below, if that occurs check if memcached is running and reachable from the server that phpMemcached was installed, one way to verify this is by doing a telnet in the memcached address on port 11211.

Points of attention that must be monitored and solutions

In the "Cache size graphic" chart you have a clear indication of the memory size allocated for it versus the consumption and waste of memory. TOTVS Fluig Platform has been designed in such a way as to never expire a record that is in the cache, but a high memory consumption can cause memcached to purge the oldest records voluntarily and so users can lose certain permissions, this behavior is evidenced in the amount of "Evictions" that the system is performing.

Memcached was designed to work with 1 megabyte blocks by default, those blocks are named Slab and group records of similar sizes. When you request memcached to store a record, a calculation is performed in Slab to know if it will be inserted: if Slab is full, a new page will be created.  In the example below, Slab 1 stores information the size of 96 bytes, as Slab page size has been parameterized to 5 megabytes an approximate math (1024*1024* 5)/96 = 54613 leads us to the conclusion that Slab contains 54613 records so if one more record is entered, memcached will create a new page for that Slab.

There are situations, as at Totvs, in which we may have groups of more than 100 thousand users, which ends up generating records larger than the ones supported by default, for those cases, we should parameterize memcached to work with larger Slabs by changing the parameter -I (“I” as in “insect”).

In the example below the page size configuration of the slab was 5 megabytes, that number is relatively high and ends up generating a big waste of memory. You can see in the example below that the maximum size for the records is 1.8 megabytes, so you can have a more assertive configuration and waste less if the parameter were close to 2 megabytes.

The waste happens when a page is created in the slab and the entire configured memory is allocated (in this case 5 megabytes): if that page no longer receives any or few records, that entire area will be underutilized.

How to fix.

The memcached parameterizations are a math that we have been unraveling slowly, nowadays we know that, in the Totvs environment, a Slab page size setting has to be larger than the standard and thus there is a greater consumption of memory, the settings are - m512-i5m, that is, we are allocating 512 megabytes of memory and 5 megabytes for Slab pages.

More information about parameterization options can be obtained on memcached sites and on that site with the description of the parameters http://linux.die.net/man/1/memcached