Simple process management in bash

Too many times I've found myself writing bash scripts responsible for spawning children processes which in turn work on a sets of files or other. They usually look cryptic.. Then I discovered 2 much nicer solutions: the built-in bash command `jobs` and the ever present `xargs`.

Running `jobs -p` in a bash shell will give you a NL separated list of the PIDs of all the children of the current shell.

Even better: let's avoid overloading the CPU of our dev machines with too many processes by limiting the number of children processes which may run in parallel dynamically based on the actual number of cores available.

#!/bin/bash

function get_cpu_count {
  echo -n $(cat /proc/cpuinfo | grep "^processor" | wc -l);
}

#!/bin/bash

function process_file {
  local FILE="${1}";
  # ...
}

function process_files_in_batches {
  local FILES="${1}";
  local CPU_COUNT=$(get_cpu_count);
  for FILE in $FILES; do
    process_file $FILE &
    while [[ $(jobs -p | wc -l) -ge $CPU_COUNT ]]; do
      sleep 0.5;
    done
  done
}

# using it becomes as easy as:
process_files_in_batches $(find /var/lib/data/ -type f);

And then there is xargs, another simple solution for making your scripts run in parallel, using the -P parameter:

function process_files_in_batches_2 {
  find /var/lib/data/ -type f -print0 | xargs --null -P $(get_cpu_count) -I{} cmd {};
}

The -print0 of `find` will make the list of matching files be separated by null bytes instead of the default NL characters. The --null of `xargs` tells `xargs` that the data is null byte separated. The -I{} defines {} as the replacement token inside of the `xargs` command to run (noted as cmd here). Finally, the -P $(get_cpu_count) defines how many processes xargs will allow to run in parallel at any given time.

That'll all there is to it, very simple parallel bash scripting. When I first discovered this I immediately re-used it for refactoring out some dirty multiprocessing attempts in various php workers. In then I feel like it somehow approached the code to that unix saying "do one thing and do it well" by allowing the php scripts to only handle their processing and not the system processes.. and all that in just a few generic shell functions ^^

How to throttle rsync

I was facing an issue where I needed to rsync a large set of small files (about half a million) from backend servers towards web fronts. That seemed simple but after a few test runs I noticed just how much load the rsync put onto the front-end, bringing the load up to 40 (on quad core VMs) so this was obviously bad.. The apache instances were suffering as a cause, connections piled up, things quickly got out of hand.

Solution? `ionice`!

Where `nice` is used for CPU scheduling `ionice` is used for io scheduling and together they can tame a massive rsync.

Here's what the final command line looked like (initiated from a backend server):

rsync --timeout=480 -z --compress-level=9 --rsync-path="nice -n19 ionice -c3 rsync" --recursive --delete-during --delete-excluded /local/path $REMOTE_SERVER:/remote/path

The magic here is in the --rsync-path parameter where we're defining the path on the remote server for rsync. Instead of using just rsync we're setting a nice'd and ionice'd rsync. Finally the -c3 parameter for ionice is stating that the io scheduling should only occur when the disk is considered idle as to avoid any blocking (especially important for the apache processes which are serving from disk!).

See more about ionice and nice.

Adventures of a backend developer

Recently I was given a rather interesting task which consisted of loading a CSV formatted file into a memcached bucket. Our original loader, which was a rough cli script written in (my beloved) PHP, just wasn't cutting it any longer. Error rates were rather high and the loading speed *suboptimal* to say the least, and lets not even talk about CPU/Mem usage. Back to the drawing board, I decided to get away from PHP and look at what the unix toolbox proposed as solutions. If you're not aware of memcached please check it out, it's a blazing fast and very high performant key-value store with a very simple ASCII protocol. I won't go into the details here but you can learn more about memcached's merits here and the protocol specification here. Out of that spec, the only piece which I was interested in was the "set" command, the command used for inserting/updating a key/value pair.

set <key> <flags> <exptime> <bytes> [noreply]\r\n
<value>\r\n

So what I have to work with is a linux server, the default toolbox, and a set of CSV files where the 3 columns represent: key, value, expiration time. Parsing CVS files with linux? That's a job for awk! Awk is amazing for parsing CSV files, it reads line by line and allows you to manipulate/conditionally compute on the values of the columns. So I decided to let awk tranform these CSV file into the memcached ASCII protocol. Half the problem is solved, I can transform a CSV file into a suite of memcached commands.. Now for the second half: how to pipe this into memcached? As soon as I hear network pipe, I think of another awesome linux tool: netcat. Netcat is a very simple tool which allows you to connect a socket to a remote machine and pipe whatever you like over the socket. All that said, here is a shortened demo of the final working solution, looking at the different steps I had taken: A snippet of an exmaple flat data file:

key_00001###value_00001###60
key_00002###value_00002###64
key_00003###value_00003###69
key_00004###value_00004_different_length_value###30

And the bash script using awk/netcat magic:

#!/bin/bash

function warmup_memcached_from_csv {
 local FILE="${1}";
 local HOST="${2}";
 local PORT="${3}";

 awk 'BEGIN { FS="###" } {
  printf "set %s 0 %i %i\r\n%s\r\n", $1, $3, length($2), $2
 } END { printf "quit\r\n" }' $FILE | netcat $HOST $PORT
}

So for each line in the csv file, 2 lines of output are generated, the memcached set commands. The first line of the example csv file would become:

set key_00001 0 60 11\r\n
value_00001\r\n

To execute the function on a set of csv files to warmup a bucket running on localhost:11211 would look something like this

for F in $(find /var/lib/csv_files/ -name '*.csv' -type -f); do
 warmup_memcached_from_csv $F 127.0.0.1 11211
done

That worked like a charm.. but there was more to the requirements. I needed to know the number of errors, if any, that occured during the sets. The memcached host will respond to the command with "ERROR\r\n" in the case of an error on a set command so all I needed was to count them. I know the tools for the job, grep and wc. Grep is a regex pattern matching filter which can work on streams and wc is the "word counter" which can also count lines.. So putting the 2 together, I'll filter out everything except error messages and then count them. Because netcat is a bidirection network pipe, this was very simple to tack on to the last implementation:

#!/bin/bash

function warmup_memcached_from_csv {
 local FILE="${1}";
 local HOST="${2}";
 local PORT="${3}";

 awk 'BEGIN { FS="###" } {
  printf "set %s 0 %i %i\r\n%s\r\n", $1, $3, length($2), $2
 } END { printf "quit\r\n" }' $FILE | netcat $HOST $PORT | grep 'ERROR' | wc -l
}

Now this function no longer spews a long list of memcached commands, it only returns a number, the error count. Using the function changes slightly to actually make use of the error counter:

for F in $(find /var/lib/csv_files/ -name '*.csv' -type -f); do
 ERROR_COUNT=$(warmup_memcached_from_csv $F 127.0.0.1 11211);
 if [ "$ERROR_COUNT" -gt 0 ]; then
  # .. error reporting! retry loops! everything is possible ..
 fi
done

Up till now everything is nice and simple.. The last requirement was bit of a mystery to me about how I would achieve at first: the values columns of these files were't actually plain text, but serialized php objects. The webservers using them were using the php-memcache extension. This is where the second parameter of the memcached set command comes into play, the records "flag". The memcached server has an extra interger flag field which is stored with the set command and retrieved with the get command but isn't actually used by the memcached server, instead it's only purpose it to have "metadata" for the memcached clients. Using the php-memcache extension it's possible to do something like so:

$m = new memcache();
$m->connect('127.0.0.1', 11211);
$m->set('key_00001', array('this', 'is', 'a', 'php', 'array'));

echo var_export( $m->get('key_00001'), true );

# which will display:
#
# array (
#   0 => 'this',
#   1 => 'is',
#   2 => 'a',
#   3 => 'php',
#   4 => 'array',
# )

So if the CSV file contained:

key_00001###a:5:{i:0;s:4:"this";i:1;s:2:"is";i:2;s:1:"a";i:3;s:3:"php";i:4;s:5:"array";}###60

After loading the file the frontend servers should be able to:

echo var_export( $m->get('key_00001'), true );
# and get the same result:
# array (
#   0 => 'this',
#   1 => 'is',
#   2 => 'a',
#   3 => 'php',
#   4 => 'array',
# )

So somehow, the php-memcache extension must be using this extra flag field internally to know what's to be unserialized after a get() and whats to be considered non-serialized data. A quick look into the inside of the extension told me exactly what I expected:

$ php --re memcache # this is an EXTREMELY usefull parameter ( --re ) of php for viewing which ini options/methods/functions/classes/interfaces/constants any extension installed exposes

And sure enough, in the constant definitions I found the one I was looking for: MEMCACHE_HAVE_SESSION. Admittingly the constants name wasn't very clear to me at first but then I realized that the method php-memcache uses for serializing/unserializing is the same that's defined for session storage .. so maybe not un/serialize(), it could just as well be defined as json encoding or maybe even some exotic XML format.. joy.

$ php --re memcache
... snippet ...
    Constant [ integer MEMCACHE_COMPRESSED ] { 2 }
    Constant [ integer MEMCACHE_HAVE_SESSION ] { 1 }
... snippet ...

Now I was ready to make the last and final change to the loader, simply setting the keys flag to 1 so that the php-memcache extension would automatically unserialize the value after issuing a get():

#!/bin/bash

function warmup_memcached_from_csv {
 local FILE="${1}";
 local HOST="${2}";
 local PORT="${3}";

 awk 'BEGIN { FS="###" } {
  printf "set %s 1 %i %i\r\n%s\r\n", $1, $3, length($2), $2
 } END { printf "quit\r\n" } ' $FILE | netcat $HOST $PORT | grep 'ERROR' | wc -l
}

Problem solved. We can now load the CSV files using only the linux toolbox and still get the values from memcached from php without having to make any change to the frontend webservers. Comments? Bug reports? Critics? They're all welcome!

Automating command line scripts in PHP

I started out creating this class once I finally got fed up with continuously looking up documentation for `expect` (and tcl, the language of `expect` scripts). I was aiming for something with the same functionality as expect but with a more familiar PHP syntax environment. After a few successful attempts I fell onto the expect pecl ( http://pecl.php.net/package/expect ) but seeing the interface exposed, I really didn't care for it either and just decided to continue this one. This is what I came up with followed by an example use case:

# namespace hl5\expect;

/**
 * this is a pure php implementation of something like 'expect'. this is for
 * automating cli applications where the apps block while waiting for user
 * input. In the end, the goal is to automate these input blocking cli apps for
 * example, subversion asking for svn username and password.
 * @since april 23rd, 2012
 * @author shean massey
 */
class proc {
  private $_proc_resource = null;
  private $_cases = array();
  private $_output_text = '';
  private $_pipes = null;

  public function __construct( $process = '' ) {
    if ( $process ) {
      $this->open( $process );
    }
  }

  /**
   * create the internal pipes for reading and writing to the opened process
   */
  public function open( $process ) {
    $descriptors = array(
      0 => array('pipe', 'r'),
      1 => array('pipe', 'w'),
      2 => array('file', '/tmp/expect_errors.log', 'a'),
    );
    $this->_proc_resource = proc_open( $process, $descriptors, $pipes );
    if ( ! is_resource( $this->_proc_resource ) ) {
      throw new \exception('proc_open() failed to create a resource');
    }
    $this->_pipes = $pipes;
    return $this;
  }

  /**
   * close the opened pipes+process and return the return code of that process
   */
  public function close() {
    fclose( $this->_pipes[0] );
    fclose( $this->_pipes[1] );
    $return_code = proc_close( $this->_proc_resource );
    return $return_code;
  }

  /**
   * set the rules, and assoc array where the keys are strings
   * of 'expected text' and the values are closures to be executed
   * once the expected string has been matched
   */
  public function on( array $cases ) {
    $this->_cases = $cases;
    return $this;
  }

  /**
   * write to the opened processes input stream
   */
  public function write( $text ) {
    fwrite( $this->_pipes[0], $text );
    return $this;
  }

  /**
   * write() + newline
   */
  public function writeln( $text ) {
    return $this->write( $text . PHP_EOL );
  }

  /**
   * compare the end of the buffer with $expected_text
   */
  private function _caught( $expected_text ) {
    $expected_length = strlen( $expected_text );
    $buffer_length = strlen( $this->_output_text );
    $search_position = $buffer_length - $expected_length;
    $test_string = substr($this->_output_text,$search_position,$expected_length);
    return ($test_string===$expected_text);
  }

  /**
   * run the script that was previously open()d and apply the expectation rules
   */
  public function run() {
    while ( $char = stream_get_contents( $this->_pipes[1], 1 ) ) {
      $this->_output_text .= $char;
      foreach ( $this->_cases as $expected_text => $closure ) {
        if ( ! $this->_caught( $expected_text ) ) continue;
        $closure();
        $this->_output_text = '';
      }
    }
  }
}

The concept is simple: open a process with proc_open and use the i/o pipes for reading the open processes output and reacting on certain output string by, for example, writing to the open processes input. A very simple use case is for subversion -update which prompts for an svn username and password:

$proc = new proc();
$proc->open('svn update /var/projects/library/');
$proc->on(array(
    'Subversion user name: ' => function() use ( $proc ) {
        $proc->writeln('shean.massey');
    },
    'Subversion password: ' => function() use ( $proc ) {
        $proc->writeln('my_super_secret_password!');
        exit( $proc->close() );
    },
))->run();

This seems pretty straight forward but just in case: I initiate a new proc() object, and open the process 'svn'. I then set 2 rules to react on, first once the script detects 'Subversion user name: ' (output from the svn update) the script sends my subversion username to svn prompt. Then once the script detects the output 'Subversion password: ' the script sends my password the closes returning the return value of the command.
This script will run on Windows! The only modification needed is changing the output file for the errors to a plausible path.

Cheers

Organizing your libraries like a boss

Recently I've been creating a lot of small PHP libs/modules that I would like to be able to reuses in other projects. Issues always come about when I need to include the files from other modules and how to keep them path independent and organized without wrapping everything into a parent namespace. Namespaces, autoloaders, and a bootloader has been the best solution I've come up with until now. So for example, I have a message queue module (group of classes) which I keep in a directory /hacklabs/modules/mq/. Everything in this directory is namespaced in 'mq', for example the class "queue" ( /hacklabs/modules/mq/queue.php ):

namespace mq;

class queue {
    # ...
}

/hacklabs/modules/mq/exceptions/empty_queue_exception.php:

namespace mq\exceptions;

class empty_queue_exception {
    # ...
}

In another project, I would like to use this mq module. Now comes the bootloader of the mq module ( /hacklabs/modules/mq/bootloader.php )

define('MQ_PATH', dirname(__FILE__));

# add the parent directory to the include path:
set_include_path( get_include_path() . PATH_SEPARATOR . realpath( dirname( __FILE__ ) . '/../' ) );

spl_autoload_register();

At this point, if I want to use the module 'mq' from a project in /hacklabs/projects/big_project/ all I need to do is include the bootloader from my modules and the default spl autoloader will take care of the rest:

namespace big_project;

require_once('/hacklabs/modules/mq/bootloader.php');
# also load a forums module bootloader:
require_once('/hacklabs/modules/forums/bootloader.php');

$queue = new \mq\queue();
$msg = new \mq\message();

try {
    $queue->send( $msg );
} catch ( \mq\exceptions\empty_queue_exception $e ) {
    # ...
} catch ( \exception $e ) {
    # ...
}

$forum = new \forums\forum();
# ... 

Using this technique, I can make module specific defines in the modules bootloader while the actual autoload include path is anything in /hacklabs/modules/*. Include just the modules bootloader, then the rest is handled via spl_autoload_register() and namespaced code.