Django admin - register all your models the quick and dirty way

Would you like to just import all your models and make them all available to your django admin interface? I would, and here's how I'm doing it now:

# app/admin.py
from django.contrib import admin

from . import models


for _, inst in models.__dict__.items():
    if isinstance(inst, type):
        try:
            admin.site.register(inst)
        except:
            pass

I needed the empty try/except block to avoid bugging out on the use of an AbstractUser model but for now, this works great.

Divide and Conquer.. with bash and friends!

Another day, another script in need of a huge performance boost. The scenerio is somewhat common to me: datasets in the form of files are being transferred using bash scripts (glorified rsync wrappers with some additional error checking) and after the transfer, the same bash process spawns a php script for the actual proccessesing of the records (in this case, performs some transformations followed by DB inserts).

The problem was that the (single threaded) php step was unable to keep up with the high rates of massive files (>3 Gb) being sent its way.

Instead of trying to optimize the php processor, I decided to wrap it with some job control logic and divide the hug files into smaller chunks. Finally, a use-case for using `split` (man 1 split).

The idea was to cut the big file into lots of smaller pieces and then spawn X php processes to consume the files in parallel. For this problem I decided to split the file based on the number of lines because each line contains a full record and then spawn off 1 php process per chunk. It worked like a charm, dividing the work into smaller and easier to digest pieces fed into a pool of php workers:

#
# divide_and_conquer( file_name, max_number_of_workers )
#
function divide_and_conquer {
 local _big_file="${1}"; shift;
 local _parallel_count="${1}"; shift;

 # where to place the file chunks:
 local _prefix="/var/tmp/$(date +%s)_";

 split --lines=10000 ${_big_file} ${_prefix};

 local _file_chunks="$(ls -1 ${_prefix}*)";

 for f in ${_file_chunks}; do
  # spawn off a php worker for this file chunk and if the php script returns a non error code,
  # then delete the processed chunk:
  ( php /var/script.php ${f} && rm ${f} ) &

  # limit the total number of worker processes:
  while [[ $(jobs -p | wc -l) -ge ${_parallel_count} ]]; do
   sleep 0.1;
  done
 done

 # wait for the last of the children:
 while [[ $(jobs -p | wc -l) -ne 0 ]]; do
  sleep 0.1;
 done
}

# and let's use it:
divide_and_conquer "/var/lib/huge_file.csv" "8"

Cheers -

php syntax checking with vim

Showing off your cowboy skills by modifying PHP code on a production server with vim? Here's a neat trick to at least check the syntax of the modified file before saving it:

:w ! php -l

Or even save the above as a binding (ctrl-b) in your vimrc:

map <C-B> :w ! php -l<CR>

And a big thanks to http://vim.wikia.com/wiki/Runtime_syntax_check_for_php for making this so clear.

ZeroMQ, HWM, and INPROC

I have been banging my head pretty hard for the past 2 days using ZeroMQ with a combination of inproc transports and HWM. In my scenario I have an inproc ZMQ_PUSH socket pushing and an inproc ZMQ_PULL reading from the pipe. The client (pusher) blocked somewhere between 1 and 2k messages and no matter what I set it's ZMQ_HWM to it just kept blocking. As the project I'm working on requires something like a dynamic HWM I wrote a custom implementation of HWM and deal with them in the application and just wanted to disable the built-in version. Before filing a bug report I decided to take a glance into the github repo.. and that's where it all made sense, here's a snippet from this source:

        // The total HWM for an inproc connection should be the sum of
        // the binder's HWM and the connector's HWM.
        int sndhwm = 0;
        if (options.sndhwm != 0 && peer.options.rcvhwm != 0)
            sndhwm = options.sndhwm + peer.options.rcvhwm;
        int rcvhwm = 0;
        if (options.rcvhwm != 0 && peer.options.sndhwm != 0)
            rcvhwm = options.rcvhwm + peer.options.sndhwm;

And it's even clearly stated in the comments just above, I need to set the HWM to 0 on both the sender AND the receiver :) Moral of the story is: if you need to set HWM limits on inproc sockets, you have to set the limits on both sides!

Locking processes with flock

Got a cronjob that might overlap if it runs slower than usual and you need to avoid multiple instances running?
Here's how to lock them using flock in bash:

function delicate_process() {
  # the 'locked down' code
  return 0;
}

function main() {
  (
    if ! flock -x --nonblock 200; then
      return 1;
    fi

    delicate_process;

  ) 200>/var/lock/.my.lock
}

main "${@}";

This is essentially opening the file /var/lock/.my.lock and assigning it the FD 200, then inside flock attempts a non blocking exclusive lock on the FD 200, returning `1` on failure.

Dynamic /etc/hosts with a simple template engine

This is a pretty niche script that probably won't do much good to anybody else.. but just in case I'm pushing it down the intertubes. I find myself editing blocks of domains in my hosts file regularly so I finally decided to save a few seconds everyday and create a template engine for my hosts file where I can now use a script to change the IP for blocks of virtual hosts all in a single command. The syntax is short and simple:

# <pool_a>
127.0.0.1 vhost-1.service_a.com service_a
127.0.0.1 vhost-2.service_a.com
# </pool_a>

# <pool_b>
192.168.0.1 vhost-1.service_b.com service_b
192.168.0.1 vhost-2.service_b.com
# </pool_b>

It uses an html-like tag inside of a bash comment for opening and closing the "blocks" of virtual hosts. Then comes the script which now reads and modifies my hosts files using these "templates":

#!/bin/bash

HOSTS_FILE='/etc/hosts';

function update_dyn_block() {
  local block_name="${1}";
  local ip_address="${2}";

  # sanity checks, only one named template block allowed:
  if [ $(cat ${HOSTS_FILE} | grep "# <${block_name}>" 2>/dev/null | wc -l) -ne 1 ]; then
    echo -n "missing or duplicate named template opening-tag found in ";
    echo "${HOSTS_FILE}: <${block_name}>";
    return 1;
  fi

  if [ $(cat ${HOSTS_FILE} | grep "# </${block_name}>" 2>/dev/null | wc -l) -ne 1 ]; then
    echo -n "missing or duplicate named template closing-tag found in ";
    echo "${HOSTS_FILE}: </${block_name}>";
    return 1;
  fi

  # get the line numbers of the line numbers of the template opening and closing tags:
  local opening_tag_at=$(cat ${HOSTS_FILE} | grep -n "# <${block_name}>" | cut -d: -f1);
  local closing_tag_at=$(cat ${HOSTS_FILE} | grep -n "# </${block_name}>" | cut -d: -f1);

  # echo "template block found between lines: ${opening_tag_at} and ${closing_tag_at}";

  local temp_file=$(mktemp);

  # ...
  cat ${HOSTS_FILE} | awk "
  {
    if ( NR > ${opening_tag_at} && NR < ${closing_tag_at} && NF >= 2 ) {
      printf \"%s\", \"${ip_address}\";
      for ( i = 2; i <= NF; i++ ) {
        printf \" %s\", \$i;
      }
      printf \"\n\";
    } else {
      print;
    }
  }" > ${temp_file};

  diff ${HOSTS_FILE} ${temp_file};

  mv ${temp_file} ${HOSTS_FILE};
}

if [ "${1}" = "" -o "${2}" = "" ]; then
  echo "usage: $0 <block_name> <new_ip_address>";
  echo;
  echo "inside of your hosts file, use 'tags' to define a block of entries with this syntax:";
  echo;
  echo "# <tag-name>";
  echo "127.0.0.1 host-1.hostname.com";
  echo "127.0.0.1 host-2.hostname.com";
  echo "# </tag-name>";
  echo;
  echo -n "note: the script is picky about the space between the hashtag and the '<' ";
  echo "character of the tag -- dont forget the space";
  exit 1;
else
  update_dyn_block "${1}" "${2}";
fi

And voila, now I can just call the script with the parameters "block name" and "new ip address" and my hosts file will be updated using the same template and print a diff showing the changes that were made to the hosts file. Here's a complete demo with the HOSTS_FILE=/tmp/hosts set in the change_hosts script:

[root@localhost requester]# cat /tmp/hosts
# <pool_a>
127.0.0.1 vhost-1.service_a.com service_a
127.0.0.1 vhost-2.service_a.com
# </pool_a>

# <pool_b>
192.168.0.1 vhost-1.service_b.com service_b
192.168.0.1 vhost-2.service_b.com
# </pool_b>
[root@localhost requester]# ~/bin/change_hosts pool_a 127.0.0.2
2,3c2,3
< 127.0.0.1 vhost-1.service_a.com service_a
< 127.0.0.1 vhost-2.service_a.com
---
> 127.0.0.2 vhost-1.service_a.com service_a
> 127.0.0.2 vhost-2.service_a.com
[root@localhost requester]# ~/bin/change_hosts pool_b 198.168.0.2
7,8c7,8
< 192.168.0.1 vhost-1.service_b.com service_b
< 192.168.0.1 vhost-2.service_b.com
---
> 198.168.0.2 vhost-1.service_b.com service_b
> 198.168.0.2 vhost-2.service_b.com
[root@localhost requester]# cat /tmp/hosts
# <pool_a>
127.0.0.2 vhost-1.service_a.com service_a
127.0.0.2 vhost-2.service_a.com
# </pool_a>

# <pool_b>
198.168.0.2 vhost-1.service_b.com service_b
198.168.0.2 vhost-2.service_b.com
# </pool_b>

Cheers

Simple process management in bash

Too many times I've found myself writing bash scripts responsible for spawning children processes which in turn work on a sets of files or other. They usually look cryptic.. Then I discovered 2 much nicer solutions: the built-in bash command `jobs` and the ever present `xargs`.

Running `jobs -p` in a bash shell will give you a NL separated list of the PIDs of all the children of the current shell.

Even better: let's avoid overloading the CPU of our dev machines with too many processes by limiting the number of children processes which may run in parallel dynamically based on the actual number of cores available.

#!/bin/bash

function get_cpu_count {
  echo -n $(cat /proc/cpuinfo | grep "^processor" | wc -l);
}

#!/bin/bash

function process_file {
  local FILE="${1}";
  # ...
}

function process_files_in_batches {
  local FILES="${1}";
  local CPU_COUNT=$(get_cpu_count);
  for FILE in $FILES; do
    process_file $FILE &
    while [[ $(jobs -p | wc -l) -ge $CPU_COUNT ]]; do
      sleep 0.5;
    done
  done
}

# using it becomes as easy as:
process_files_in_batches $(find /var/lib/data/ -type f);

And then there is xargs, another simple solution for making your scripts run in parallel, using the -P parameter:

function process_files_in_batches_2 {
  find /var/lib/data/ -type f -print0 | xargs --null -P $(get_cpu_count) -I{} cmd {};
}

The -print0 of `find` will make the list of matching files be separated by null bytes instead of the default NL characters. The --null of `xargs` tells `xargs` that the data is null byte separated. The -I{} defines {} as the replacement token inside of the `xargs` command to run (noted as cmd here). Finally, the -P $(get_cpu_count) defines how many processes xargs will allow to run in parallel at any given time.

That'll all there is to it, very simple parallel bash scripting. When I first discovered this I immediately re-used it for refactoring out some dirty multiprocessing attempts in various php workers. In then I feel like it somehow approached the code to that unix saying "do one thing and do it well" by allowing the php scripts to only handle their processing and not the system processes.. and all that in just a few generic shell functions ^^

How to throttle rsync

I was facing an issue where I needed to rsync a large set of small files (about half a million) from backend servers towards web fronts. That seemed simple but after a few test runs I noticed just how much load the rsync put onto the front-end, bringing the load up to 40 (on quad core VMs) so this was obviously bad.. The apache instances were suffering as a cause, connections piled up, things quickly got out of hand.

Solution? `ionice`!

Where `nice` is used for CPU scheduling `ionice` is used for io scheduling and together they can tame a massive rsync.

Here's what the final command line looked like (initiated from a backend server):

rsync --timeout=480 -z --compress-level=9 --rsync-path="nice -n19 ionice -c3 rsync" --recursive --delete-during --delete-excluded /local/path $REMOTE_SERVER:/remote/path

The magic here is in the --rsync-path parameter where we're defining the path on the remote server for rsync. Instead of using just rsync we're setting a nice'd and ionice'd rsync. Finally the -c3 parameter for ionice is stating that the io scheduling should only occur when the disk is considered idle as to avoid any blocking (especially important for the apache processes which are serving from disk!).

See more about ionice and nice.

Adventures of a backend developer

Recently I was given a rather interesting task which consisted of loading a CSV formatted file into a memcached bucket. Our original loader, which was a rough cli script written in (my beloved) PHP, just wasn't cutting it any longer. Error rates were rather high and the loading speed *suboptimal* to say the least, and lets not even talk about CPU/Mem usage. Back to the drawing board, I decided to get away from PHP and look at what the unix toolbox proposed as solutions. If you're not aware of memcached please check it out, it's a blazing fast and very high performant key-value store with a very simple ASCII protocol. I won't go into the details here but you can learn more about memcached's merits here and the protocol specification here. Out of that spec, the only piece which I was interested in was the "set" command, the command used for inserting/updating a key/value pair.

set <key> <flags> <exptime> <bytes> [noreply]\r\n
<value>\r\n

So what I have to work with is a linux server, the default toolbox, and a set of CSV files where the 3 columns represent: key, value, expiration time. Parsing CVS files with linux? That's a job for awk! Awk is amazing for parsing CSV files, it reads line by line and allows you to manipulate/conditionally compute on the values of the columns. So I decided to let awk tranform these CSV file into the memcached ASCII protocol. Half the problem is solved, I can transform a CSV file into a suite of memcached commands.. Now for the second half: how to pipe this into memcached? As soon as I hear network pipe, I think of another awesome linux tool: netcat. Netcat is a very simple tool which allows you to connect a socket to a remote machine and pipe whatever you like over the socket. All that said, here is a shortened demo of the final working solution, looking at the different steps I had taken: A snippet of an exmaple flat data file:

key_00001###value_00001###60
key_00002###value_00002###64
key_00003###value_00003###69
key_00004###value_00004_different_length_value###30

And the bash script using awk/netcat magic:

#!/bin/bash

function warmup_memcached_from_csv {
 local FILE="${1}";
 local HOST="${2}";
 local PORT="${3}";

 awk 'BEGIN { FS="###" } {
  printf "set %s 0 %i %i\r\n%s\r\n", $1, $3, length($2), $2
 } END { printf "quit\r\n" }' $FILE | netcat $HOST $PORT
}

So for each line in the csv file, 2 lines of output are generated, the memcached set commands. The first line of the example csv file would become:

set key_00001 0 60 11\r\n
value_00001\r\n

To execute the function on a set of csv files to warmup a bucket running on localhost:11211 would look something like this

for F in $(find /var/lib/csv_files/ -name '*.csv' -type -f); do
 warmup_memcached_from_csv $F 127.0.0.1 11211
done

That worked like a charm.. but there was more to the requirements. I needed to know the number of errors, if any, that occured during the sets. The memcached host will respond to the command with "ERROR\r\n" in the case of an error on a set command so all I needed was to count them. I know the tools for the job, grep and wc. Grep is a regex pattern matching filter which can work on streams and wc is the "word counter" which can also count lines.. So putting the 2 together, I'll filter out everything except error messages and then count them. Because netcat is a bidirection network pipe, this was very simple to tack on to the last implementation:

#!/bin/bash

function warmup_memcached_from_csv {
 local FILE="${1}";
 local HOST="${2}";
 local PORT="${3}";

 awk 'BEGIN { FS="###" } {
  printf "set %s 0 %i %i\r\n%s\r\n", $1, $3, length($2), $2
 } END { printf "quit\r\n" }' $FILE | netcat $HOST $PORT | grep 'ERROR' | wc -l
}

Now this function no longer spews a long list of memcached commands, it only returns a number, the error count. Using the function changes slightly to actually make use of the error counter:

for F in $(find /var/lib/csv_files/ -name '*.csv' -type -f); do
 ERROR_COUNT=$(warmup_memcached_from_csv $F 127.0.0.1 11211);
 if [ "$ERROR_COUNT" -gt 0 ]; then
  # .. error reporting! retry loops! everything is possible ..
 fi
done

Up till now everything is nice and simple.. The last requirement was bit of a mystery to me about how I would achieve at first: the values columns of these files were't actually plain text, but serialized php objects. The webservers using them were using the php-memcache extension. This is where the second parameter of the memcached set command comes into play, the records "flag". The memcached server has an extra interger flag field which is stored with the set command and retrieved with the get command but isn't actually used by the memcached server, instead it's only purpose it to have "metadata" for the memcached clients. Using the php-memcache extension it's possible to do something like so:

$m = new memcache();
$m->connect('127.0.0.1', 11211);
$m->set('key_00001', array('this', 'is', 'a', 'php', 'array'));

echo var_export( $m->get('key_00001'), true );

# which will display:
#
# array (
#   0 => 'this',
#   1 => 'is',
#   2 => 'a',
#   3 => 'php',
#   4 => 'array',
# )

So if the CSV file contained:

key_00001###a:5:{i:0;s:4:"this";i:1;s:2:"is";i:2;s:1:"a";i:3;s:3:"php";i:4;s:5:"array";}###60

After loading the file the frontend servers should be able to:

echo var_export( $m->get('key_00001'), true );
# and get the same result:
# array (
#   0 => 'this',
#   1 => 'is',
#   2 => 'a',
#   3 => 'php',
#   4 => 'array',
# )

So somehow, the php-memcache extension must be using this extra flag field internally to know what's to be unserialized after a get() and whats to be considered non-serialized data. A quick look into the inside of the extension told me exactly what I expected:

$ php --re memcache # this is an EXTREMELY usefull parameter ( --re ) of php for viewing which ini options/methods/functions/classes/interfaces/constants any extension installed exposes

And sure enough, in the constant definitions I found the one I was looking for: MEMCACHE_HAVE_SESSION. Admittingly the constants name wasn't very clear to me at first but then I realized that the method php-memcache uses for serializing/unserializing is the same that's defined for session storage .. so maybe not un/serialize(), it could just as well be defined as json encoding or maybe even some exotic XML format.. joy.

$ php --re memcache
... snippet ...
    Constant [ integer MEMCACHE_COMPRESSED ] { 2 }
    Constant [ integer MEMCACHE_HAVE_SESSION ] { 1 }
... snippet ...

Now I was ready to make the last and final change to the loader, simply setting the keys flag to 1 so that the php-memcache extension would automatically unserialize the value after issuing a get():

#!/bin/bash

function warmup_memcached_from_csv {
 local FILE="${1}";
 local HOST="${2}";
 local PORT="${3}";

 awk 'BEGIN { FS="###" } {
  printf "set %s 1 %i %i\r\n%s\r\n", $1, $3, length($2), $2
 } END { printf "quit\r\n" } ' $FILE | netcat $HOST $PORT | grep 'ERROR' | wc -l
}

Problem solved. We can now load the CSV files using only the linux toolbox and still get the values from memcached from php without having to make any change to the frontend webservers. Comments? Bug reports? Critics? They're all welcome!

Automating command line scripts in PHP

I started out creating this class once I finally got fed up with continuously looking up documentation for `expect` (and tcl, the language of `expect` scripts). I was aiming for something with the same functionality as expect but with a more familiar PHP syntax environment. After a few successful attempts I fell onto the expect pecl ( http://pecl.php.net/package/expect ) but seeing the interface exposed, I really didn't care for it either and just decided to continue this one. This is what I came up with followed by an example use case:

# namespace hl5\expect;

/**
 * this is a pure php implementation of something like 'expect'. this is for
 * automating cli applications where the apps block while waiting for user
 * input. In the end, the goal is to automate these input blocking cli apps for
 * example, subversion asking for svn username and password.
 * @since april 23rd, 2012
 * @author shean massey
 */
class proc {
  private $_proc_resource = null;
  private $_cases = array();
  private $_output_text = '';
  private $_pipes = null;

  public function __construct( $process = '' ) {
    if ( $process ) {
      $this->open( $process );
    }
  }

  /**
   * create the internal pipes for reading and writing to the opened process
   */
  public function open( $process ) {
    $descriptors = array(
      0 => array('pipe', 'r'),
      1 => array('pipe', 'w'),
      2 => array('file', '/tmp/expect_errors.log', 'a'),
    );
    $this->_proc_resource = proc_open( $process, $descriptors, $pipes );
    if ( ! is_resource( $this->_proc_resource ) ) {
      throw new \exception('proc_open() failed to create a resource');
    }
    $this->_pipes = $pipes;
    return $this;
  }

  /**
   * close the opened pipes+process and return the return code of that process
   */
  public function close() {
    fclose( $this->_pipes[0] );
    fclose( $this->_pipes[1] );
    $return_code = proc_close( $this->_proc_resource );
    return $return_code;
  }

  /**
   * set the rules, and assoc array where the keys are strings
   * of 'expected text' and the values are closures to be executed
   * once the expected string has been matched
   */
  public function on( array $cases ) {
    $this->_cases = $cases;
    return $this;
  }

  /**
   * write to the opened processes input stream
   */
  public function write( $text ) {
    fwrite( $this->_pipes[0], $text );
    return $this;
  }

  /**
   * write() + newline
   */
  public function writeln( $text ) {
    return $this->write( $text . PHP_EOL );
  }

  /**
   * compare the end of the buffer with $expected_text
   */
  private function _caught( $expected_text ) {
    $expected_length = strlen( $expected_text );
    $buffer_length = strlen( $this->_output_text );
    $search_position = $buffer_length - $expected_length;
    $test_string = substr($this->_output_text,$search_position,$expected_length);
    return ($test_string===$expected_text);
  }

  /**
   * run the script that was previously open()d and apply the expectation rules
   */
  public function run() {
    while ( $char = stream_get_contents( $this->_pipes[1], 1 ) ) {
      $this->_output_text .= $char;
      foreach ( $this->_cases as $expected_text => $closure ) {
        if ( ! $this->_caught( $expected_text ) ) continue;
        $closure();
        $this->_output_text = '';
      }
    }
  }
}

The concept is simple: open a process with proc_open and use the i/o pipes for reading the open processes output and reacting on certain output string by, for example, writing to the open processes input. A very simple use case is for subversion -update which prompts for an svn username and password:

$proc = new proc();
$proc->open('svn update /var/projects/library/');
$proc->on(array(
    'Subversion user name: ' => function() use ( $proc ) {
        $proc->writeln('shean.massey');
    },
    'Subversion password: ' => function() use ( $proc ) {
        $proc->writeln('my_super_secret_password!');
        exit( $proc->close() );
    },
))->run();

This seems pretty straight forward but just in case: I initiate a new proc() object, and open the process 'svn'. I then set 2 rules to react on, first once the script detects 'Subversion user name: ' (output from the svn update) the script sends my subversion username to svn prompt. Then once the script detects the output 'Subversion password: ' the script sends my password the closes returning the return value of the command.
This script will run on Windows! The only modification needed is changing the output file for the errors to a plausible path.

Cheers

Organizing your libraries like a boss

Recently I've been creating a lot of small PHP libs/modules that I would like to be able to reuses in other projects. Issues always come about when I need to include the files from other modules and how to keep them path independent and organized without wrapping everything into a parent namespace. Namespaces, autoloaders, and a bootloader has been the best solution I've come up with until now. So for example, I have a message queue module (group of classes) which I keep in a directory /hacklabs/modules/mq/. Everything in this directory is namespaced in 'mq', for example the class "queue" ( /hacklabs/modules/mq/queue.php ):

namespace mq;

class queue {
    # ...
}

/hacklabs/modules/mq/exceptions/empty_queue_exception.php:

namespace mq\exceptions;

class empty_queue_exception {
    # ...
}

In another project, I would like to use this mq module. Now comes the bootloader of the mq module ( /hacklabs/modules/mq/bootloader.php )

define('MQ_PATH', dirname(__FILE__));

# add the parent directory to the include path:
set_include_path( get_include_path() . PATH_SEPARATOR . realpath( dirname( __FILE__ ) . '/../' ) );

spl_autoload_register();

At this point, if I want to use the module 'mq' from a project in /hacklabs/projects/big_project/ all I need to do is include the bootloader from my modules and the default spl autoloader will take care of the rest:

namespace big_project;

require_once('/hacklabs/modules/mq/bootloader.php');
# also load a forums module bootloader:
require_once('/hacklabs/modules/forums/bootloader.php');

$queue = new \mq\queue();
$msg = new \mq\message();

try {
    $queue->send( $msg );
} catch ( \mq\exceptions\empty_queue_exception $e ) {
    # ...
} catch ( \exception $e ) {
    # ...
}

$forum = new \forums\forum();
# ... 

Using this technique, I can make module specific defines in the modules bootloader while the actual autoload include path is anything in /hacklabs/modules/*. Include just the modules bootloader, then the rest is handled via spl_autoload_register() and namespaced code.

A process manager in PHP

namespace shell;

/**
 * pcntl is a process manager for spawning new child processes
 * either through php closures or shell commands. pcntl will
 * make sure zombies dont start invading your systems
 * @author shean massey
 * @since Jan 7th 2012
 */
class pcntl {
    protected $_child_pids = array();
    protected $_max_wait = 0;
    protected $_debug_mode = false;

    /**
     * register the wait function as a shutdown procedure
     * and register the signal handlers for SIGCHLD and SIGTERM
     */
    public function __construct() {
        foreach ( array( SIGCHLD, SIGTERM ) as $signal )
        if ( false === ( pcntl_signal( $signal, array( $this, 'signal_handler') ) ) ) {
            throw new \exception('failed tp attach signal: ' . $signal . ' handler');
        }
        # on shutdown, wait() for all/any forked child processes 
        register_shutdown_function( array( $this, 'wait' ) );
    }

    /**
     * on destruction wait() for all/any forked child processes
     */
    public function __destruct() {
        $this->wait();
    }

    /**
     * enable/disable debug messages
     */
    public function use_debugger( $bool = true ) {
        $this->_debug_mode = (bool)$bool;
        return $this;
    }

    /**
     * this is the signal handler registered for cleaning the dead children
     * processes and remove them from the pid collection.
     */
    public function signal_handler( $signum ) {
        switch ( $signum ) {
            # this event is sent to processes when one of their child processes
            # passed away
            case SIGCHLD:
                while ( $pid = pcntl_wait( $status, WNOHANG) ) {
                    $this->debug( function() use ($pid, $status){
                        echo 'caught pid by signal handler: ', $pid, ' ';
                        echo 'return status: ', $status, ' ';
                        echo '[my pid = ', posix_getpid(), ']', PHP_EOL;
                    });
                    # there are no more children to handle:
                    if ( empty( $this->_child_pids ) ) return;
                    # pcntl_wait failed:
                    if ( $pid === -1 ) {
                        pcntl_signal_dispatch();
                        break;
                    }
                    # how could this even happen ?
                    if ( ! array_key_exists( $pid, $this->_child_pids ) ) {
                        throw new \appcore\exception('caught someone elses dead baby');
                    }
                    # remove the child pid from the pid collection
                    unset( $this->_child_pids[ $pid ] );
                }
            break;
            # this process is sent from a kill -1:
            case SIGTERM:
                
            break;
        }
    }

    /**
     * fork a closure as a new process
     */
    public function fork( \closure $function ) {
        switch ( $pid = pcntl_fork() ) {
            # error:
            case -1:
                throw new \appcore\exception('failed to fork()');
            break;
            # child proc:
            case 0:
                # empty the array of children:
                $this->_child_pids = array();
                $function();
                exit(0);
            break;
            # parent proc:
            default:
                $this->_child_pids[ $pid ] = $pid;
                $this->debug( function() use ($pid) {
                    echo 'new child: ', $pid, PHP_EOL;
                });
                pcntl_signal_dispatch();
                return $this;
            break;
        }
    }

    /**
     * fork a shell command
     */
    public function shell_fork( $cmd_line = '' ) {
        $this->fork( function() use ( $cmd_line ) {
            $cmd_line .= ' 2>&1 > /tmp/proc_manager.lck.'.posix_getpid().' &';
            shell_exec( $cmd_line );
        });
        pcntl_signal_dispatch();
        return $this;
    }

    /**
     * if the debug_mode is set, the execute the function being
     * passed as a closure. Note that this function should never
     * attempt to change the current state.
     */
    public function debug( \closure $function ) {
        if ( ! $this->_debug_mode ) return false;
        $function();
        return true;
    }

    /**
     * this will loop with short sleeps and dispatch any lingering
     * signals to the signal handler (to reap the dead children)
     */
    public function wait() {
        $this->debug( function(){
            echo 'dispatch loop', PHP_EOL;
        });
        while ( true ) {
            if ( ! $this->_child_pids ) break;
            pcntl_signal_dispatch();
            usleep(10);
        }
        return $this;
    }
}

Using it is VERY simple, you fork a new process either as a closures or as shell commands:

#!/usr/bin/env php
use_debugger( true );

$proc_manager->fork( function( ) {
  $i = 0;
  while ( $i++ < 5 ) {
    sleep( 1 );
    file_put_contents('/tmp/test', 'i = '.$i.PHP_EOL, FILE_APPEND);
  }
});

$proc_manager->fork( function( ) {
  $m = 0; while ( $m++ < 6 ) {
    sleep( 1 ); 
    file_put_contents('/tmp/test', 'm = '.$m.PHP_EOL, FILE_APPEND);
  }
});

$proc_manager->shell_fork('sleep 1 && ls -la');

$proc_manager->shell_fork('rsync # ...');

PHP: errors/warnings/notices to exceptions

Something to append to all your bootstraps:

set_error_handler( function( $num, $msg, $file, $line ) {
  # take into account the '@' operators ( or remove this line and ignore them ):
  if ( error_reporting() === 0 ) return false;
  throw new \ErrorException( $msg, $num, 0, $file, $line );
});

This will transform all user catchable internal php errors (not parse errors) into \ErrorException exceptions (a subclass of \Exception ).
Cheers

Lazy loading with PHP magic methods

Using some php magic for a dead simple lazy loading framework:

class DemoContext {
    protected $_lazy_vars = array(
        'cache' => null,
        'request' => null,
    );

    # the magic happens here ;)
    public function __get( $varname ) {
        if ( ! array_key_exists( $varname, $this->_lazy_vars ) )
            return null;
        if ( $this->_lazy_vars[ $varname ] !== null )
            return $this->_lazy_vars[ $varname ];

        $value = null;
        switch ( $varname ) {
            case 'cache':
                $value = new \appcore\cache();
            break;
            case 'request':
                $value = new \appcore\request();
            break;
        }
        $this->_lazy_vars[ $varname ] = $value;
        return $value;
    }
}

# using it:
$demo = new DemoContext();
if ( $demo->cache->check() ) {
    # ...
}

if ( $demo->request->hasHeader('Location') ) {
    # ...
}

Using lazy loading is great for setting back of any object creation until you're sure you actually need it. This could be a huge memory improvement in some cases, anywhere where there might be useless object creation.

FCache Revisited

After some refactoring I found a way to use a closure for my content cacher instead of fake "code blocks". Now using it is much easier to read and understand, the code that needs to be cached is passed as the implementation of an anonymous function. The new cache_block() method:

/**
 * cache a block of code
 * @param string $key the cache key
 * @param int $seconds the number of seconds before a cache file is outdated
 * @param function the closure to cache the contents of
 */
public static function cache_block( $key, $seconds = null, $function ) {
  if ( self::is_cached( $key, $seconds ) ) {
    echo file_get_contents( self::get_path( $key ) );
    return true;
  }
  self::init_buffer();
  call_user_func( $function );
  echo self::save_buffer( $key );
  return true;
}

And an example usage of the new method:

public function demo() {
  $self = $this; # php5.3 cant pass $this with use(), but 5.4 will be able to!

  \appcore\fcache::cache_block('demo_cache_1', 2, function() use ($self) {
    $self->show('header');
    $self->show('index');
    \appcore\fcache::cache_block('demo_cache_2', 4, function() use ($self) {
      $self->show('footer');
    });
  });
}

This creates 2 cached elements, 'demo_cache_1' and 'demo_cache_2'. The demo_cache_1 element expires after 2 seconds and the demo_cache_2 after 4 seconds. The means when the contents of the demo_cache_1 expire and the block is re-executed, the contents of demo_cache_2 may be re-used if they havn't expired yet in the creation of the outer block cache. Here's the entire fcache class refactored:

namespace appcore;

/**
 * file caching methods. this is a key/value implementation.
 * @author smassey
 * @since may 5th 2011
 * may 22nd 2011 - added events
 * oct 16th 2011 - refactored + added the cache_block() method
 */
class fcache extends namespace\base\object {
  private function __construct() {}
  private function __clone() {}

  /**
   * check if a file is cached and optionaly non expired
   * @param mixed $key
   * @param int $seconds
   * @return false if a cached file for the given key isnt found
   * or if the file exits and is outdated. return true otherwise. 
   */
  public static function is_cached( $key, $seconds = null ) {
    if ( ! file_exists( self::get_path( $key ) ) ) return false;
    return ( $seconds ) ? ( ! self::is_outdated( $key, $seconds ) ) : true;
  }

  /**
   * get a cached file
   * @param mixed $key the key of the cached file
   * @param int $exp_seconds the expire time of the cached file in seconds
   */
  public static function get_cache( $key ) {
    if (self::is_cached( $key )) {
      \appcore\events::send('trace', 'found cache ' . md5($key));
      return file_get_contents( self::get_path( $key ) );
    }
    return false;
  }

  /**
   * save contents into a cached file
   * @param $key the cache key
   * @param $value the contents to cache
   */
  public static function cache( $key, $value ) {
    $path = self::get_path( $key );
    if ( file_put_contents( $path, $value ) === false ) {
      throw new \Exception("failed to write file: $path");
    }
  }

  /**
   * cache a block of code
   * @param string $key the cache key
   * @param int $seconds the number of seconds before a cache file is outdated
   * @param function the closure to cache the contents of
   */
  public static function cache_block( $key, $seconds = null, $function ) {
    if ( self::is_cached( $key, $seconds ) ) {
      echo file_get_contents( self::get_path( $key ) );
      return true;
    }
    self::init_buffer();
    call_user_func( $function );
    echo self::save_buffer( $key );
    return true;
  }

  /**
   * start the buffers
   */
  public static function init_buffer() {
    ob_start();
  }

  /**
   * save a buffer
   * @param $key the cache key
   */
  public static function save_buffer( $key ) {
    \appcore\events::send('trace', 'saving buffer to file cache');
    $contents = ob_get_contents();
    ob_end_clean();
    self::cache( $key, $contents );
    return $contents;
  }

  /**
   * determine the real full path + filename for a cache file
   * @param string $key the cache key
   * @returns the full path + filename
   */
  protected static function get_path( $key ) {
    return CACHE_PATH . md5($key) . '.cache.php';
  }
 
  /**
   * deterine if a file is outdated
   * @param string $key the cached key
   * @param int $seconds number of seconds to check the caches age against
   */
  protected static function is_outdated( $key, $seconds = false ) {
    if ( ! $seconds ) return false;
    $file_stats = stat( self::get_path( $key ) );
    if ( ( time() - $file_stats['mtime'] ) > $seconds ) return true;
    return false;
  }
}

An expression parser in PHP

This is a little mathematical expression parser I wrote out of sheer boredom. After finding my "calculator assignment" from highschool (which worked, but had absolutly no style, just a couple hundred lines of sequential handling) I decided I really needed to take another shot at it.. And this time, to do thing right and build an expression tree from the input then evaluate the tree.

First a brief walkthrough:
- create a stack of context handlers (a handler must react on passed tokens)
- first step is to tokenize the content into a list of tokens
- push the first (global) context onto the context stack
- iterate through the token list and send the token to the last context handler on the stack
( the context handlers are responsible for pushing/poping contexts onto the stack )
- pop off the last context, this should be the first pushed, the global (the trunk of the tree)

Example input: 3 + ( 4 * 2.3 ) - 1
the tokenized list would be: ['3', '+', '(', '4', '*', '2.3', ')', '-', '1']
- push a global scope onto the context handler stack (context 0)
looping through the tokens:
on '3': the context handler adds 3 to its expression list (context 1)
on '+': the context handler adds + to its expression list (context 1)
on '(': the context handler pushes a new context onto the the context handler stack (context 2)
on '4': the context handler adds 4 to its expression list (context 2)
on '*': the context handler adds * to its expression list (context 2)
on '2.3': the context handler adds 2.3 to its expression list (context 2)
on ')': the context handler pops off 'context 2', adds it to the expression list of context 1
on '-': the context handler adds - to the expression list (context 1)
on '1': the context handler adds 1 to the expression list (context 1)

context 1 looks like this: 3 + (context 2) - 1
context 2 looks like this: 4 * 2.3

Now that the data is parsed into the tree structure, some recursive evaluation and we've got the result.

/exprlib/Parser.php

<?php
namespace exprlib;

/**
 * this model handles the tokenizing, the context stack functions, and
 * the parsing (token list to tree trans).
 * as well as an evaluate method which delegates to the global scopes evaluate.
 */

class Parser {
    protected $_content = null;
    protected $_context_stack = array();
    protected $_tree = null;
    protected $_tokens = array();

    public function __construct($content = null) {
        if ( $content ) {
        	$this->set_content( $content );
        }
    }

    /**
     * this function does some simple syntax cleaning:
     * - removes all spaces
     * - replaces '**' by '^'
     * then it runs a regex to split the contents into tokens. the set
     * of possible tokens in this case is predefined to numbers (ints of floats)
     * math operators (*, -, +, /, **, ^) and parentheses.
     */
    public function tokenize() {
        $this->_content = str_replace(array("\n","\r","\t"," "), '', $this->_content);
        $this->_content = str_replace('**', '^', $this->_content);
        $this->_content = str_replace('PI', (string)PI(), $this->_content);
        $this->_tokens = preg_split(
        	'@([\d\.]+)|(sin\(|cos\(|tan\(|sqrt\(|\+|\-|\*|/|\^|\(|\))@',
        	$this->_content,
        	null,
        	PREG_SPLIT_DELIM_CAPTURE | PREG_SPLIT_NO_EMPTY
        );
        return $this;
    }

    /**
     * this is the the loop that transforms the tokens array into
     * a tree structure.
     */
    public function parse() {
        # this is the global scope which will contain the entire tree
        $this->push_context( new \exprlib\contexts\Scope() );
        foreach ( $this->_tokens as $token ) {
        	# get the last context model from the context stack,
        	# and have it handle the next token	 
        	$this->get_context()->handle_token( $token );
        }
        $this->_tree = $this->pop_context();

        return $this;
    }

    public function evaluate() {
        if ( ! $this->_tree ) {
        	throw new \exprlib\exceptions\ParseTreeNotFoundException();
        }
        return $this->_tree->evaluate();
    }

    /*** accessors and mutators ***/

    public function get_tree() {
        return $this->_tree;
    }
    
    public function set_content($content = null) {
        $this->_content = $content;
        return $this;
    }

    public function get_tokens() {
        return $this->_tokens;
    }


    /*******************************************************
     * the context stack functions. for the stack im using 
     * an array with the functions array_push, array_pop,
     * and end to push, pop, and get the current element
     * from the stack.
     *******************************************************/

    public function push_context( \exprlib\contexts\IfContext $context ) {
        array_push( $this->_context_stack, $context );
        $this->get_context()->set_builder( $this );
    }

    public function pop_context() {
        return array_pop( $this->_context_stack );
    }

    public function get_context() {
        return end( $this->_context_stack );
    }
}

The global scope in this case is also the parent scope of all other scopes:

/exprlib/contexts/Scope.php

<?php
namespace exprlib\contexts;

class Scope implements namespace\IfContext {
    protected $_builder = null;
    protected $_children_contexts = array();
    protected $_raw_content = array();
    protected $_operations = array();

    const T_NUMBER = 1;
    const T_OPERATOR = 2;
    const T_SCOPE_OPEN = 3;
    const T_SCOPE_CLOSE = 4;
    const T_SIN_SCOPE_OPEN = 5;
    const T_COS_SCOPE_OPEN = 6;
    const T_TAN_SCOPE_OPEN = 7;
    const T_SQRT_SCOPE_OPEN = 8;

    public function set_builder( \exprlib\Parser $builder ) {
        $this->_builder = $builder;
    }

    public function __toString() {
        return implode('', $this->_raw_content);
    }

    public function add_operation( $operation ) {
        $this->_operations[] = $operation;
    }

    /**
     * handle the next token from the tokenized list. example actions
     * on a token would be to add it to the current context expression list,
     * to push a new context on the the context stack, or pop a context off the
     * stack.
     */
    public function handle_token( $token ) {
        $type = null;
    
        if ( in_array( $token, array('*','/','+','-','^') ) ) $type = self::T_OPERATOR;
        if ( $token === ')' ) $type = self::T_SCOPE_CLOSE;
        if ( $token === '(' ) $type = self::T_SCOPE_OPEN;
        if ( $token === 'sin(' ) $type = self::T_SIN_SCOPE_OPEN;
        if ( $token === 'cos(' ) $type = self::T_COS_SCOPE_OPEN;
        if ( $token === 'tan(' ) $type = self::T_TAN_SCOPE_OPEN;
        if ( $token === 'sqrt(' ) $type = self::T_SQRT_SCOPE_OPEN;

        if ( is_null( $type ) ) {
            if ( is_numeric( $token ) ) {
                $type = self::T_NUMBER;
                $token = (float)$token;
            }
        }

        switch ( $type ) {
            case self::T_NUMBER:
            case self::T_OPERATOR:
                $this->_operations[] = $token;
            break;
            case self::T_SCOPE_OPEN:
                $this->_builder->push_context( new namespace\Scope() );
            break;
            case self::T_SIN_SCOPE_OPEN:
                $this->_builder->push_context( new namespace\SineScope() );
            break;
            case self::T_COS_SCOPE_OPEN:
                $this->_builder->push_context( new namespace\CosineScope() );
            break;
            case self::T_TAN_SCOPE_OPEN:
                $this->_builder->push_context( new namespace\TangentScope() );
            break;
            case self::T_SQRT_SCOPE_OPEN:
                $this->_builder->push_context( new namespace\SqrtScope() );
            break;
            case self::T_SCOPE_CLOSE:
                $scope_operation = $this->_builder->pop_context();
                $new_context = $this->_builder->get_context();
                if ( is_null( $scope_operation ) || ( ! $new_context ) ) {
                    # this means there are more closing parentheses than openning
                    throw new \exprlib\exceptions\OutOfScopeException();
                }
                $new_context->add_operation( $scope_operation );
            break;
            default:
                throw new \exprlib\exceptions\UnknownTokenException($token);
            break;
        }
    }

    /**
     * order of operations:
     * - parentheses, these should all ready be executed before this method is called
     * - exponents, first order
     * - mult/divi, second order
     * - addi/subt, third order
     */
    protected function _expression_loop( & $operation_list ) {
        while ( list( $i, $operation ) = each ( $operation_list ) ) {
            if ( ! in_array( $operation, array('^','*','/','+','-') ) ) continue;

            $left =  isset( $operation_list[ $i - 1 ] ) ? (float)$operation_list[ $i - 1 ] : null;
            $right = isset( $operation_list[ $i + 1 ] ) ? (float)$operation_list[ $i + 1 ] : null;

            # if ( is_null( $left ) || is_null( $right ) ) throw new \Exception('syntax error');

            $first_order = ( in_array('^', $operation_list) );
            $second_order = ( in_array('*', $operation_list ) || in_array('/', $operation_list ) );
            $third_order = ( in_array('-', $operation_list ) || in_array('+', $operation_list ) );

            $remove_sides = true;
            if ( $first_order ) {
                switch( $operation ) {
                    case '^': $operation_list[ $i ] = pow( (float)$left, (float)$right ); break;
                    default: $remove_sides = false; break;
                }
            } elseif ( $second_order ) {
                switch ( $operation ) {
                    case '*': $operation_list[ $i ] = (float)($left * $right); break;
                    case '/': $operation_list[ $i ] = (float)($left / $right); break;
                    default: $remove_sides = false; break;
                }
            } elseif ( $third_order ) {
                switch ( $operation ) {
                    case '+': $operation_list[ $i ] = (float)($left + $right); break;
                    case '-': $operation_list[ $i ] = (float)($left - $right); break;
                    default: $remove_sides = false; break;
                }
            }

            if ( $remove_sides ) {
                unset( $operation_list[ $i + 1 ], $operation_list[ $i - 1 ] );
                reset( $operation_list = array_values( $operation_list ) );
            }
        }
        if ( count( $operation_list ) === 1 ) return end( $operation_list );
        return false;
    }

    # order of operations:
    # - sub scopes first
    # - multiplication, division
    # - addition, subtraction
    # evaluating all the sub scopes (recursivly):
    public function evaluate() {
        foreach ( $this->_operations as $i => $operation ) {
            if ( is_object( $operation ) ) {
                $this->_operations[ $i ] = $operation->evaluate();
            }
        }

        $operation_list = $this->_operations;

        while ( true ) {
            $operation_check = $operation_list;
            $result = $this->_expression_loop( $operation_list );
            if ( $result !== false ) return $result;
            if ( $operation_check === $operation_list ) {
                break;
            } else {
                reset( $operation_list = array_values( $operation_list ) );
            }
        }
        throw new \Exception('failed... here');
    }
}

Here are 2 of the scopes that extends the global (parent) scope: the sine and squared root scopes

/exprlib/contexts/SineScope.php

<?php
namespace exprlib\contexts;

class SineScope extends namespace\Scope {
    public function evaluate() {
        return sin( deg2rad( parent::evaluate() ) );
    }
}

\exprlib\contexts\SqrtScope.php

<?php
namespace exprlib\contexts;

class SqrtScope extends namespace\Scope {
	public function evaluate() {
		return sqrt( parent::evaluate() );
	}
}

To conclude, here is a user interface I built for the testing. It's a command line shell which inputs expressions and outputs the evaluated expressions or thrown exception messages in case of error:

#!/usr/bin/php
<?php
include('exprlib/loaders.php');

$builder = new \exprlib\Parser();

while ( (fputs(STDOUT,'math > ')) && $e = fgets(STDIN) ) {
	if ( ! ($e = trim($e)) ) continue;
	if ( in_array( $e, array('quit','exit',':q') ) ) break;

	try {
		$result = $builder->set_content($e)->tokenize()->parse()->evaluate();
	} catch ( \exprlib\exceptions\UnknownTokenException $exception ) {
		echo 'unknown token exception thrown in expression: ', $e, PHP_EOL;
		echo 'token: "',$exception->getMessage(),'"',PHP_EOL;
		continue;
	} catch ( \exprlib\exceptions\ParseTreeNotFoundException $exception ) {
		echo 'parse tree not found (missing content): ', $e, PHP_EOL;
		continue;
	} catch ( \exprlib\exceptions\OutOfScopeException $exception ) {
		echo 'out of scope exception thrown in: ', $e, PHP_EOL;
		echo 'you should probably count your parentheses', PHP_EOL;
		continue;
	} catch ( \Exception $exception ) {
		echo 'unknown exception thrown: ', $e, PHP_EOL;
		echo $exception->getMessage(), PHP_EOL;
		continue;
	}

	echo $result, PHP_EOL;
}

How to kill an HTTP connection and continue processing (PHP and Apache2)

Today I fell onto yet another really neat trick with HTTP: closing the connection without closing the server side process that's handling the request. It's actually a really simple trick and only involves sending the right headers to the browser on which the browser cuts the connection. The magic headers:

Content-length: 0
Connection: close

And example of using this with PHP:

<?php
header("Connection: close");
header("Content-length: 0");
flush(); # flush will send the headers we just defined,

# from here on out, the browser has (at least should have) closed the connection,
# here is where we get to do all the time taking tasks without blocking the users browser
file_put_contents('/tmp/demo.txt', time(), FILE_APPEND);
sleep(5);
file_put_contents('/tmp/demo.txt', time(), FILE_APPEND);

In this case we're not actually sending any kind of result/output to the browser other than the headers to direct it to kill the connection. Another possibility would be using this together with content buffering so we can send an actual response and set the correct headers:

<?php
ob_start();

echo "<html><head></head><body>";
echo "<h1>this is a demo</h1>";
echo "</body></html>";

header("Connection: close");
header("Content-length: " . (string)ob_get_length());

ob_end_flush();
ob_flush(); 
flush();

sleep(5);
# work, work and more work... 
# send a mail, log actions to files, databases, ... all that really slow stuff ^^
file_put_contents('/tmp/tmp_OB', 'this is a test');

Multiple Java JARs in a JAR

After looking around for a way to compile the dependant jars of my java app into a single java jar with my app, I finally found the solution here http://download.oracle.com/javase/tutorial/deployment/jar/downman.html

So for context, I was playing with JOGL (JSR-231) and found myself having to append the jogl.jar and gluegen-rt.jar to the classpath to compile and run my toy app. I finally decided to do it the right way and throw everything into a single Jar file for simplicity and ease of distribution. Thankfully the jar archiver tool rocks and made this a lot simpler than the other solutions I read online about using OneJar.

My Manifest.txt file looks like this:

Manifest-Version: 1.0
Main-Class: App
Class-Path: extralibs/gluegen-rt.jar extralibs/jogl.jar

My file layout looks like this:

/src/
  - App.java
  - mylibs/
    - MyClasses.java
  - extralibs/
    - jogl.jar
    - gluegen-rt.jar
    - Manifest.txt

And finally the jar line that create the jar file from my app sources and the extra jogl jars:

jar cfm Exec.jar extralibs/Manifest.txt App.class mylibs/*.class extralibs/*.jar

So now by running that, I get a single JAR archive that contains my application and the dependant jars with their correct classpaths :)

Regards

ArchLinux and .pacnew files

After a system update with using pacman on ArchLinux, you will probably have some core configuration files that are out of date. Pacman handles this by creating new config files and appending .pacnew to their name. Here's a simple script to find all the pacnew files and merge them with the your current config files using meld:

for PACNEW_FILE in `find /etc/ -name "*.pacnew"`; do
    BASE_FILE=`echo $PACNEW_FILE | sed '/\.pacnew//'`;
    meld $PACNEW_FILE $BASE_FILE;
    echo -n "Remove the file $PACNEW_FILE ? [y|n] ";
    read -n 1 CHOICE
    if [ "$CHOICE" = "y" ]; then
        rm $PACNEW_FILE;
    fi
done;

MySQL LIMIT clause.. and when you shouldn't use it

Today I learned something that just amazed me about my trusty MySQL and most of all, the fact that I'm only learning this today, after years of using MySQL.

The senerio: a coworker of mine was running a query on a MySQL table that held around 10 million records. For context: the table was using the MyISAM engine and was not being modified in any way (no updates/inserts, only selects). The table also had an auto-incremented indexed ID field. The query itsself was very simple:

select id_content, content from `tablename` limit 1, 100000;

The limit offset was actually being used as a sliding window, and the limit count was a constant 100 thousand. Watching the the mysql server status in real time showed something strange, as the script was running locks started appearing and blocking as well as slowing down dramatically for each new window.

Reason: The limit clause. Little did we know, the MySQL limit has a "bizarre" implementation that doesn't work like I would have expected, when using an offset/count pair, the server actually selects ALL the records up until the offset, then selects `offset` more records and returns the later.

So for example, when do a "select * from tablename limit 100, 10;" the server is actually selecting the 110 first results then only returning the last 10 of that result set. Not too bad for a small table, but for a larger table this is just horrible: "Limit 100000, 10" actually selects (and allocates memory for) 100010 records then using the result of that, returns the last 10.

Better solution: stick with the indexed field and use where clauses instead.

select id_content, content from `tablename` where id_content > 10000 and id_content < 100010;

Cheers!

Edit*

I would like to clarify that using the LIMIT clause shouldn't be banned altogther, simply the limit clauses that use an offset and on large tables ;)