Thursday, December 6, 2012

Simple process management in bash

Too many times I've found myself writing bash scripts responsible for spawning children processes which in turn work on a sets of files or other. They usually look cryptic.. Then I discovered 2 much nicer solutions: the built-in bash command `jobs` and the ever present `xargs`.

Running `jobs -p` in a bash shell will give you a NL separated list of the PIDs of all the children of the current shell.

Even better: let's avoid overloading the CPU of our dev machines with too many processes by limiting the number of children processes which may run in parallel dynamically based on the actual number of cores available.


function get_cpu_count {
  echo -n $(cat /proc/cpuinfo | grep "^processor" | wc -l);


function process_file {
  local FILE="${1}";
  # ...

function process_files_in_batches {
  local FILES="${1}";
  local CPU_COUNT=$(get_cpu_count);
  for FILE in $FILES; do
    process_file $FILE &
    while [[ $(jobs -p | wc -l) -ge $CPU_COUNT ]]; do
      sleep 0.5;

# using it becomes as easy as:
process_files_in_batches $(find /var/lib/data/ -type f);

And then there is xargs, another simple solution for making your scripts run in parallel, using the -P parameter:
function process_files_in_batches_2 {
  find /var/lib/data/ -type f -print0 | xargs --null -P $(get_cpu_count) -I{} cmd {};

The -print0 of `find` will make the list of matching files be separated by null bytes instead of the default NL characters. The --null of `xargs` tells `xargs` that the data is null byte separated. The -I{} defines {} as the replacement token inside of the `xargs` command to run (noted as cmd here). Finally, the -P $(get_cpu_count) defines how many processes xargs will allow to run in parallel at any given time.

That'll all there is to it, very simple parallel bash scripting. When I first discovered this I immediately re-used it for refactoring out some dirty multiprocessing attempts in various php workers. In then I feel like it somehow approached the code to that unix saying "do one thing and do it well" by allowing the php scripts to only handle their processing and not the system processes.. and all that in just a few generic shell functions ^^