“There are only two hard things in Computer Science: cache invalidation and naming things.” — Phil Karlton
Mr. Karlton was not wrong. In my day-to-day job, cache invalidation is something that can easily disrupt releases – the fact that the result from an API is cached can be easily forgotten, for example. It has often caused us to pause and re-evaluate exactly what our applications are doing. Some of our APIs are quite static in nature and are cached appropriately (general rule of thumb is that more static data is cached for longer periods of time). When it comes to update these APIs, there are often many layers of cache to bust through in order to prune stale data. If we forget to clear one cache, the stale data can again propagate to all the other caches in the stash. Not cool.
Not unlike an onion, this can definitely cause tears.
Memcache & Redis
Invalidating keys in redis is relatively simple via redis-cli:
redis-cli KEYS "session:*" | xargs redis-cli DEL
memcached on the other hand does not support namespaced deletes, nor does it have a tool to interact with the server. The only real way to interact with the server is via telnet or similar tool via TCP/IP (such as nc). This caused a desire to write a tool to invalidate cache quickly so that I could test these problem APIs more effectively. Below is the source code (a bash script) – it requires netcat to be installed and within your path.
#!/bin/bash TIMEOUT=2 SERVERS=("127.0.0.1:11211") DEFAULTPORT=11211 KEYLIMIT=100 function usage { echo "" echo "$0 [regex]" echo "Used to invalidate keys on memcached servers" echo "" } function memcache_netcat { netcat -q $TIMEOUT $SERVER $PORT } function memcache_delete { echo "DELETING: $1" RESULT=$(echo "delete $1" | memcache_netcat) } # Parameter is required if [ -z $1 ]; then usage exit 1 fi # For each server... (in SERVER:PORT format) for definition in "${SERVERS[@]}" do IFS=':' read -ra server <<< "$definition" SERVER=${server[0]} PORT=${server[1]-$DEFAULTPORT} LOOPS=0 echo "" echo "Invalidating keys on: $SERVER:$PORT" echo "Searching for : $1" echo "" echo "stats items" | memcache_netcat | while read line; do LOOPS=$[$LOOPS+1] let "ITERS=$LOOPS % 10" # Each STATS ITEM row brings back 10 statistics. Skip all but first row. if [ $ITERS -eq 1 ]; then read -ra chunks <<< "$line" # If this not a STATS ITEMS row, skip it if [ ${#chunks[@]} -lt 3 ]; then continue fi SLAB=${chunks[1]} XIFS=$IFS IFS=" " # Search this slab for the keys it contains echo "stats cachedump $SLAB $KEYLIMIT" | memcache_netcat | while read row; do # If the key matches the search, delete KEY=`echo "$row" | tr -d '\b\r' | sed 's/^.\{4\} \([^ ]*\).*$/\1/'` if [[ $KEY = "END" ]]; then continue fi if [[ $KEY =~ $1 ]]; then read -ra parts <<< "$row" memcache_delete $KEY fi done IFS=$XIFS fi done done echo "DONE." echo "" [/sourcecode]<div class="wp-git-embed" style="margin-bottom:10px; border:1px solid #CCC; text-align:right; width:99%; margin-top:-13px; font-size:11px; font-style:italic;"><span style="display:inline-block; padding:4px;">memcache_invalidator</span><a style="display:inline-block; padding:4px 6px;" href="https://raw.github.com/jonnu/tools/master/memcache_invalidator" target="_blank">view raw</a><a style="display:inline-block; padding:4px 6px; float:left;" href="https://github.com/jonnu/tools/blob/master/memcache_invalidator" target="_blank">view file on <strong>GitHub</strong></a></div> <h3>Usage example</h3> To use this script, make sure it is executable and pass a regular expression in as the first argument. As an example, let's invalidate all keys that start with <em>session:</em> chmod a+x memcache_invalidator ./memcache_invalidator ^session
Here is some sample output, showing that we have deleted two keys (that I added for testing purposes) from local memcache:
jonnu@onion:$ ./memcache_invalidator ^session Invalidating keys on: 127.0.0.1:11211 Searching for : ^session DELETING: session_9fc9575c7eb47fbcdb39c2a872ea74d8 DELETING: session_2bdace452a1904970c457f7ddfd6a132 DONE
Suggestions on how to improve this tool are welcome – either comment here or just fork & send me a merge request on github.