Memcache
Summary
First, you start up the memcached daemon on as many spare machines as you have. The daemon has no configuration file, just a few command line options, only 3 or 4 of which you'll likely use:
./memcached -d -m 2048 -l 10.0.0.40 -p 11211
This starts memcached up as a daemon, using 2GB of memory, and listening on IP 10.0.0.40, port 11211. Because a 32-bit process can only address 4GB of virtual memory (usually significantly less, depending on your operating system), if you have a 32-bit server with 4-64GB of memory using PAE you can just run multiple processes on the machine, each using 2 or 3GB of memory.
What are some limits in memcached I might hit?: [1]
The simple limits you will probably see with memcache are the key and item size limits. Keys are restricted to 250 characters. Stored data cannot exceed 1 megabyte in size, since that is the largest typical slab size.
Installation
Yum Method
RPMForge Method: (version 1.4.5-1.el5.rf as of 2010.10.25)
- See RPMforge for RPMforge installation
- Yum method info provided here: http://serverfault.com/questions/342826/problems-installing-memcached
yum install memcached --enablerepo=rpmforge --enablerepo=rpmforge-extras # x64 only - to fix x64 issue with SSLeay: # wget http://pkgs.repoforge.org/perl-Net-SSLeay/perl-Net-SSLeay-1.36-1.el5.rfx.x86_64.rpm # rpm -Uvh perl-Net-SSLeay-1.36-1.el5.rfx.x86_64.rpm
# to fix: Starting memcached: chown: cannot access `/var/run/memcached': No such file or directory # touch /var/run/memcached
/etc/sysconfig/memcached (original):
PORT="11211" USER="nobody" MAXCONN="1024" CACHESIZE="64" OPTIONS=""
/etc/sysconfig/memcached (modified, with cachesize in megabytes):
PORT="11211" USER="nobody" MAXCONN="1024" # in megabytes: CACHESIZE="128" # additional options, specify address to listen on OPTIONS="-l 127.0.0.1"
Start service:
chkconfig memcached on service memcached start
- ?eh? yum install php-eaccelerator
Manual Method
Download Libevent:
mkdir -p ~/.src ; cd ~/.src wget http://www.monkey.org/~provos/libevent-1.3e.tar.gz tar -zvxf libevent-1.3e.tar.gz cd libevent-1.3e ./configure; make; make install;
#32bit: ln -s /usr/local/lib/libevent-1.3e.so.1 /lib/libevent-1.3e.so.1 #64bit: ln -s /usr/local/lib/libevent-1.3e.so.1 /lib64/libevent-1.3e.so.1
Download memcached:
mkdir -p ~/.src ; cd ~/.src wget http://www.danga.com/memcached/dist/memcached-1.2.2.tar.gz tar -zxvf memcached-1.2.2.tar.gz cd memcached-1.2.2 ./configure; make; make install;
Start memcached:
memcached -d -u nobody -m 512 127.0.0.1 -p 11211
---
Often libevent.so cannot be found when executing memcache. A useful command LD_DEBUG, is very helpful to determine where libraries are being loaded from.
# help LD_DEBUG=help memcached -v LD_DEBUG=libs memcached -v 2>&1 > /dev/null | less 18990: find library=libevent-1.3b.so.1 [0]; searching ... 18990: trying file=/usr/lib/libevent-1.3b.so.1 18990: memcached: error while loading shared libraries: libevent-1.3b.so.1: cannot open shared object file: No such file or directory
Simply place the library where memcached will find it and execute memcached:
ln -s /usr/local/lib/libevent-1.3e.so.1 /lib/libevent-1.3e.so.1 #64bit: ln -s /usr/local/lib/libevent-1.3e.so.1 /lib64/libevent-1.3e.so.1
Testing memcache
How to install memcache on Debian Etch
Quick stats check:
echo -e "stats\nquit" | nc 127.0.0.1 11211
Testing the memcached daemon with telnet:
telnet 1.2.3.4 11211
The commands look like this:
set <key> <flag> <exptime> <bytes>\r\n
Example:
set test2 1 0 2 ab STORED
get test2 VALUE test2 1 2 ab END
set test4 1 0 2 abcde CLIENT_ERROR bad data chunk ERROR
To see statistics:
stats
To exit:
quit
The flag is an arbitrary number that you can use in your client logic. It is intended to be metadata that you can assign to each cached object.
I set the exptime to zero (never expire)
the bytes to the number of characters I want to store. If the bytes and the number of characters don't match, an error occurs.
For other commands see protocol.txt
An example of how to get the stats using a bash script
Stats Check
Dump all stats:
echo -e "stats\nquit" | nc 127.0.0.1 11211
Get important items:
echo -e "stats\nquit" | nc 127.0.0.1 11211 | grep -e "\(uptime\|get_hits\|get_misses\|bytes \|limit_maxbytes\)" # uptime - Number of seconds this server has been running # get_hits - request that are in cache # get_misses - requests that weren't in cache # limit_maxbytes - total max bytes can store # bytes - current bytes in cache
Stats Protocol
Name Type Meaning
----------------------------------
pid 32u Process id of this server process
uptime 32u Number of seconds this server has been running
time 32u current UNIX time according to the server
version string Version string of this server
pointer_size 32 Default size of pointers on the host OS
(generally 32 or 64)
rusage_user 32u:32u Accumulated user time for this process
(seconds:microseconds)
rusage_system 32u:32u Accumulated system time for this process
(seconds:microseconds)
curr_items 32u Current number of items stored by the server
total_items 32u Total number of items stored by this server
ever since it started
bytes 64u Current number of bytes used by this server
to store items
curr_connections 32u Number of open connections
total_connections 32u Total number of connections opened since
the server started running
connection_structures 32u Number of connection structures allocated
by the server
cmd_get 64u Cumulative number of retrieval requests
cmd_set 64u Cumulative number of storage requests
get_hits 64u Number of keys that have been requested and
found present
get_misses 64u Number of items that have been requested
and not found
evictions 64u Number of valid items removed from cache
to free memory for new items
bytes_read 64u Total number of bytes read by this server
from network
bytes_written 64u Total number of bytes sent by this server to
network
limit_maxbytes 32u Number of bytes this server is allowed to
use for storage.
threads 32u Number of worker threads requested.
(see doc/threads.txt)
Source: Protocol - http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt
Slab Size
What are some limits in memcached I might hit?:
- "The simple limits you will probably see with memcache are the key and item size limits. Keys are restricted to 250 characters. Stored data cannot exceed 1 megabyte in size, since that is the largest typical slab size."
- "Actually, it's a compile time option. The default is to use the internal slab allocator. You really really want to use the built-in slab allocator. At first memcached did just use malloc/free for everything. However this does not play very well with OS memory managers. You get fragmentation, and your OS ends up spending more time trying to find contiguous blocks of memory to feed malloc() than it does running the memcached process. If you disagree, of course you're free to try malloc! just don't complain on the lists ;)
- The slab allocator was built to work around this. Memory is allocated in chunks internally and constantly reused. Since memory is broken into different size slabs, you do waste memory if your items do not fit perfectly into the slab the server chooses to put it in. This has enjoyed considerable efficiency improvements by Steven Grimm.
- Some older posts about the slab changes (power of n vs power of 2), and some tradeoffs are on the mailing list:
- http://lists.danga.com/pipermail/memcached/2006-May/002163.html
- http://lists.danga.com/pipermail/memcached/2007-March/003753.html
- And if you'd like to attempt to use malloc/free and see how it works, you may define 'USE_SYSTEM_MALLOC' in build process. It might not be tested very well, so getting developer support for it is unlikely."
References:
- "The maximum object size is a few bytes under 1MB. You're trying to set about 250k more than it can take."
- "You can recompile memcached with a higher POWER_LARGEST if you need more than 1MB. It's in slabs.c at the top. Make it 21 to get 2MB max, or 22 for 4MB max, etc."
- "The maximum size is 1 megabyte. It can be increased; I believe there was a patch (against the old 1.1.12 version) to do it. The trick is that memcached's memory manager allocates memory in 1MB chunks, so increasing the maximum is not just a matter of changing a "maximum object size" setting somewhere. But it is certainly doable."
- "I propose all objects are stored as a linked list of 64 byte chunks."
- "We've talked internally about eventually moving away from the slab-class-based system. It should be possible to run a job in the background (especially in the multithreaded version, but even in the single-threaded one) to shuffle objects around in memory to pack them optimally. If you separate the fixed-size headers from the item data, you can even do away with byte-alignment-based waste. That would give you 100% memory efficiency for any arbitrary mix of item sizes, and true LRU behavior across the whole cache, at the cost of greater CPU consumption."
- Patch - Configurable slab size and large object support
- memcached overrides memory limit
- Patch - Increased memory efficiency
Startup Scripts
touch /etc/memcached.conf cat > /etc/memcached.conf
#Memory a usar -m 16 # default port -p 11211 # user to run daemon nobody/apache/www-data -u nobody # only listen locally -l 127.0.0.1
touch /etc/init.d/memcached chmod +x /etc/init.d/memcached cat > /etc/init.d/memcached
#!/bin/bash
#
# memcached This shell script takes care of starting and stopping
# standalone memcached.
#
# chkconfig: - 80 12
# description: memcached is a high-performance, distributed memory
# object caching system, generic in nature, but
# intended for use in speeding up dynamic web
# applications by alleviating database load.
# processname: memcached
# config: /etc/memcached.conf
# Source function library.
. /etc/rc.d/init.d/functions
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/memcached
DAEMONBOOTSTRAP=/usr/local/bin/start-memcached
DAEMONCONF=/etc/memcached.conf
NAME=memcached
DESC=memcached
PIDFILE=/var/run/$NAME.pid
[ -x $DAEMON ] || exit 0
[ -x $DAEMONBOOTSTRAP ] || exit 0
RETVAL=0
start() {
echo -n $"Starting $DESC: "
daemon $DAEMONBOOTSTRAP $DAEMONCONF
RETVAL=$?
[ $RETVAL -eq 0 ] && touch $PIDFILE
echo
return $RETVAL
}
stop() {
echo -n $"Shutting down $DESC: "
killproc $NAME
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f $PIDFILE
return $RETVAL
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
restart|reload)
stop
start
RETVAL=$?
;;
status)
status $prog
RETVAL=$?
;;
*)
echo $"Usage: $0 {start|stop|restart|status}"
exit 1
esac
exit $RETVAL
touch /usr/local/bin/start-memcached chmod +x /usr/local/bin/start-memcached cat > /usr/local/bin/start-memcached
#!/usr/bin/perl -w
# start-memcached
# 2003/2004 - Jay Bonci <jaybonci@debian.org>
# This script handles the parsing of the /etc/memcached.conf file
# and was originally created for the Debian distribution.
# Anyone may use this little script under the same terms as
# memcached itself.
use strict;
if ($> != 0 and $< != 0) {
print STDERR "Only root wants to run start-memcached.\n";
exit;
}
my $etcfile = shift || "/etc/memcached.conf";
my $params = [];
my $etchandle;
# This script assumes that memcached is located at /usr/bin/memcached, and
# that the pidfile is writable at /var/run/memcached.pid
my $memcached = "/usr/local/bin/memcached";
my $pidfile = "/var/run/memcached.pid";
# If we don't get a valid logfile parameter in the /etc/memcached.conf file,
# we'll just throw away all of our in-daemon output. We need to re-tie it so
# that non-bash shells will not hang on logout. Thanks to Michael Renner for
# the tip
my $fd_reopened = "/dev/null";
sub handle_logfile {
my ($logfile) = @_;
$fd_reopened = $logfile;
}
sub reopen_logfile {
my ($logfile) = @_;
open *STDERR, ">>$logfile";
open *STDOUT, ">>$logfile";
open *STDIN, ">>/dev/null";
$fd_reopened = $logfile;
}
# This is set up in place here to support other non -[a-z] directives
my $conf_directives = {
"logfile" => \&handle_logfile
};
if (open $etchandle, $etcfile) {
foreach my $line (<$etchandle>) {
$line =~ s/\#.*//go;
$line = join ' ', split ' ', $line;
next unless $line;
next if $line =~ /^\-[dh]/o;
if ($line =~ /^[^\-]/o) {
my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/;
$conf_directives->{$directive}->($arg);
next;
}
push @$params, $line;
}
}
unshift @$params, "-u root" unless (grep $_ eq '-u', @$params);
$params = join " ", @$params;
if (-e $pidfile) {
open PIDHANDLE, "$pidfile";
my $localpid = <PIDHANDLE>;
close PIDHANDLE;
chomp $localpid;
if (-d "/proc/$localpid") {
print STDERR "memcached is already running.\n";
exit;
} else {
`rm -f $localpid`;
}
}
my $pid = fork();
if ($pid == 0) {
reopen_logfile($fd_reopened);
exec "$memcached $params";
exit(0);
} elsif (open PIDHANDLE,">$pidfile") {
print PIDHANDLE $pid;
close PIDHANDLE;
} else {
print STDERR "Can't write pidfile to $pidfile.\n";
}
service memcached restart chkconfig memcached on
An alternative startup script:
Bash Stat Command
#!/bin/bash
#
# Make a 'stat' command on a memcache daemon server
# Outputs to a pipe for later using with syslog-ng
#
# alexandre.abreu at gmail.com
PATH=/bin:/usr/bin:/sbin:/usr/sbin
SERVER=localhost
PORT=11111
FIFO=/export/logs/memcache.fifo
BUFF="memcache: "
[ -p $FIFO ] || mkfifo $FIFO || exit 1
exec 666<>/dev/tcp/$SERVER/$PORT || exit 1
append() {
BUFF="$BUFF $@"
[ "$1" == "uptime" ] && RUNTIME=`echo "scale=3; $2/60" | bc`
[ "$1" == "get_hits" ] && {
rate=`echo "scale=3; $2/$RUNTIME" | bc`
rate=`printf "%.3f" $rate`
append hitrate $rate
}
[ "$1" == "get_misses" ] && {
rate=`echo "scale=3; $2/$RUNTIME" | bc`
rate=`printf "%.3f" $rate`
append missrate $rate
}
}
echo "stats" >&666
while read line; do
line=`echo $line | sed 's|[[:cntrl:]]||g'`
set - $line
append $2 $3
[ "$2" == "limit_maxbytes" ] && break
done <&666
echo "$BUFF" > $FIFO
echo "quit" >&666
exit 0
My Version:
#!/bin/bash
#
# Make a 'stat' command on a memcache daemon server
# Outputs to a pipe for later using with syslog-ng
#
PATH=/bin:/usr/bin:/sbin:/usr/sbin
SERVER=10.20.30.40
PORT=11211
FIFO=/tmp/memcache.fifo
[ -p $FIFO ] || mkfifo $FIFO || exit 1
exec 666<>/dev/tcp/$SERVER/$PORT || exit 1
echo "stats" >&666
while read line; do
echo $line;
FLAG=`echo $line | cut -b 1-3`
[ "$FLAG" == "END" ] && break
done <&666
echo "quit" >&666
exit 0