code. hack it.: July 2011

Tuesday, July 19, 2011

How to kill an HTTP connection and continue processing (PHP and Apache2)

Today I fell onto yet another really neat trick with HTTP: closing the connection without closing the server side process that's handling the request. It's actually a really simple trick and only involves sending the right headers to the browser on which the browser cuts the connection. The magic headers:

Content-length: 0
Connection: close

And example of using this with PHP:

<?php
header("Connection: close");
header("Content-length: 0");
flush(); # flush will send the headers we just defined,

# from here on out, the browser has (at least should have) closed the connection,
# here is where we get to do all the time taking tasks without blocking the users browser
file_put_contents('/tmp/demo.txt', time(), FILE_APPEND);
sleep(5);
file_put_contents('/tmp/demo.txt', time(), FILE_APPEND);

In this case we're not actually sending any kind of result/output to the browser other than the headers to direct it to kill the connection. Another possibility would be using this together with content buffering so we can send an actual response and set the correct headers:

<?php
ob_start();

echo "<html><head></head><body>";
echo "<h1>this is a demo</h1>";
echo "</body></html>";

header("Connection: close");
header("Content-length: " . (string)ob_get_length());

ob_end_flush();
ob_flush(); 
flush();

sleep(5);
# work, work and more work... 
# send a mail, log actions to files, databases, ... all that really slow stuff ^^
file_put_contents('/tmp/tmp_OB', 'this is a test');

Sunday, July 17, 2011

Multiple Java JARs in a JAR

After looking around for a way to compile the dependant jars of my java app into a single java jar with my app, I finally found the solution here http://download.oracle.com/javase/tutorial/deployment/jar/downman.html

So for context, I was playing with JOGL (JSR-231) and found myself having to append the jogl.jar and gluegen-rt.jar to the classpath to compile and run my toy app. I finally decided to do it the right way and throw everything into a single Jar file for simplicity and ease of distribution. Thankfully the jar archiver tool rocks and made this a lot simpler than the other solutions I read online about using OneJar.

My Manifest.txt file looks like this:

Manifest-Version: 1.0
Main-Class: App
Class-Path: extralibs/gluegen-rt.jar extralibs/jogl.jar

My file layout looks like this:

/src/
  - App.java
  - mylibs/
    - MyClasses.java
  - extralibs/
    - jogl.jar
    - gluegen-rt.jar
    - Manifest.txt

And finally the jar line that create the jar file from my app sources and the extra jogl jars:

jar cfm Exec.jar extralibs/Manifest.txt App.class mylibs/*.class extralibs/*.jar

So now by running that, I get a single JAR archive that contains my application and the dependant jars with their correct classpaths :)

Regards

Saturday, July 16, 2011

ArchLinux and .pacnew files

After a system update with using pacman on ArchLinux, you will probably have some core configuration files that are out of date. Pacman handles this by creating new config files and appending .pacnew to their name. Here's a simple script to find all the pacnew files and merge them with the your current config files using meld:

for PACNEW_FILE in `find /etc/ -name "*.pacnew"`; do
    BASE_FILE=`echo $PACNEW_FILE | sed '/\.pacnew//'`;
    meld $PACNEW_FILE $BASE_FILE;
    echo -n "Remove the file $PACNEW_FILE ? [y|n] ";
    read -n 1 CHOICE
    if [ "$CHOICE" = "y" ]; then
        rm $PACNEW_FILE;
    fi
done;

Thursday, July 7, 2011

MySQL LIMIT clause.. and when you shouldn't use it

Today I learned something that just amazed me about my trusty MySQL and most of all, the fact that I'm only learning this today, after years of using MySQL.

The senerio: a coworker of mine was running a query on a MySQL table that held around 10 million records. For context: the table was using the MyISAM engine and was not being modified in any way (no updates/inserts, only selects). The table also had an auto-incremented indexed ID field. The query itsself was very simple:

select id_content, content from `tablename` limit 1, 100000;

The limit offset was actually being used as a sliding window, and the limit count was a constant 100 thousand. Watching the the mysql server status in real time showed something strange, as the script was running locks started appearing and blocking as well as slowing down dramatically for each new window.

Reason: The limit clause. Little did we know, the MySQL limit has a "bizarre" implementation that doesn't work like I would have expected, when using an offset/count pair, the server actually selects ALL the records up until the offset, then selects `offset` more records and returns the later.

So for example, when do a "select * from tablename limit 100, 10;" the server is actually selecting the 110 first results then only returning the last 10 of that result set. Not too bad for a small table, but for a larger table this is just horrible: "Limit 100000, 10" actually selects (and allocates memory for) 100010 records then using the result of that, returns the last 10.

Better solution: stick with the indexed field and use where clauses instead.

select id_content, content from `tablename` where id_content > 10000 and id_content < 100010;

Cheers!

Edit*

I would like to clarify that using the LIMIT clause shouldn't be banned altogther, simply the limit clauses that use an offset and on large tables ;)

code. hack it.