for PACNEW_FILE in `find /etc/ -name "*.pacnew"`; do
BASE_FILE=`echo $PACNEW_FILE | sed '/\.pacnew//'`;
meld $PACNEW_FILE $BASE_FILE;
echo -n "Remove the file $PACNEW_FILE ? [y|n] ";
read -n 1 CHOICE
if [ "$CHOICE" = "y" ]; then
rm $PACNEW_FILE;
fi
done;
Saturday, July 16, 2011
ArchLinux and .pacnew files
After a system update with using pacman on ArchLinux, you will probably have some core configuration files that are out of date. Pacman handles this by creating new config files and appending .pacnew to their name. Here's a simple script to find all the pacnew files and merge them with the your current config files using meld:
Thursday, July 7, 2011
MySQL LIMIT clause.. and when you shouldn't use it
Today I learned something that just amazed me about my trusty MySQL and most of all, the fact that I'm only learning this today, after years of using MySQL.
The senerio: a coworker of mine was running a query on a MySQL table that held around 10 million records. For context: the table was using the MyISAM engine and was not being modified in any way (no updates/inserts, only selects). The table also had an auto-incremented indexed ID field. The query itsself was very simple:
The limit offset was actually being used as a sliding window, and the limit count was a constant 100 thousand. Watching the the mysql server status in real time showed something strange, as the script was running locks started appearing and blocking as well as slowing down dramatically for each new window.
Reason: The limit clause. Little did we know, the MySQL limit has a "bizarre" implementation that doesn't work like I would have expected, when using an offset/count pair, the server actually selects ALL the records up until the offset, then selects `offset` more records and returns the later.
So for example, when do a "select * from tablename limit 100, 10;" the server is actually selecting the 110 first results then only returning the last 10 of that result set. Not too bad for a small table, but for a larger table this is just horrible: "Limit 100000, 10" actually selects (and allocates memory for) 100010 records then using the result of that, returns the last 10.
Better solution: stick with the indexed field and use where clauses instead.
Cheers!
Edit*
I would like to clarify that using the LIMIT clause shouldn't be banned altogther, simply the limit clauses that use an offset and on large tables ;)
The senerio: a coworker of mine was running a query on a MySQL table that held around 10 million records. For context: the table was using the MyISAM engine and was not being modified in any way (no updates/inserts, only selects). The table also had an auto-incremented indexed ID field. The query itsself was very simple:
select id_content, content from `tablename` limit 1, 100000;
The limit offset was actually being used as a sliding window, and the limit count was a constant 100 thousand. Watching the the mysql server status in real time showed something strange, as the script was running locks started appearing and blocking as well as slowing down dramatically for each new window.
Reason: The limit clause. Little did we know, the MySQL limit has a "bizarre" implementation that doesn't work like I would have expected, when using an offset/count pair, the server actually selects ALL the records up until the offset, then selects `offset` more records and returns the later.
So for example, when do a "select * from tablename limit 100, 10;" the server is actually selecting the 110 first results then only returning the last 10 of that result set. Not too bad for a small table, but for a larger table this is just horrible: "Limit 100000, 10" actually selects (and allocates memory for) 100010 records then using the result of that, returns the last 10.
Better solution: stick with the indexed field and use where clauses instead.
select id_content, content from `tablename` where id_content > 10000 and id_content < 100010;
Cheers!
Edit*
I would like to clarify that using the LIMIT clause shouldn't be banned altogther, simply the limit clauses that use an offset and on large tables ;)
Saturday, June 18, 2011
a file caching soultion for php
I was looking for a file caching system that could cache blocks of php code output in a way that would dramatically reduce database hits. I came up with a static container, that:
- may start/stop a php output buffer
- saves the output buffer to a file
- uses a key/value design, keys correspond to hashed file names
- can check if a certain block of code is buffered by a key
- can use an expire parameter (on lookup, not save)
CACHE_PATH should be defined as the path to a dir, with the leading slash / and needs read+write permission.
Using it now, heres an example of how to use the file caching in a controller:
Not the best use of it, but simplicity is pretty.
Check if theres a cache saved labeled "page" that is less than 300 seconds old. If found, then show the contents of that cache. Else do some intensive work, heavy db queries and save the all the outputed results to a cache named "page".
- may start/stop a php output buffer
- saves the output buffer to a file
- uses a key/value design, keys correspond to hashed file names
- can check if a certain block of code is buffered by a key
- can use an expire parameter (on lookup, not save)
namespace appcore;
/**
* file caching methods.
*/
class fcache extends namespace\base\object {
private function __construct() {}
private function __clone() {}
/**
* check if a file is cached and optionaly non expired
* @param mixed $key
* @param int $exp_seconds number of seconds the file can exist
* before considered experied
*/
public static function is_cached( $key, $exp_seconds = null ) {
$is_valid = true;
if (!file_exists(self::get_path( $key ))) return false;
if ( is_int( $exp_seconds ) ):
$file_stats = stat( self::get_path( $key ) );
if ((time()-$file_stats['mtime'])>$exp_seconds):
\appcore\events::send('trace', 'cached file has expired');
$is_valid = false;
endif;
endif;
return $is_valid;
}
/**
* get a cached file
* @param mixed $key the key of the cached file
* @param int $exp_seconds the expire time of the cached file in seconds
*/
public static function get_cache( $key ) {
if (self::is_cached( $key )):
\appcore\events::send('trace', 'found cache ' . md5($key));
return file_get_contents( self::get_path( $key ) );
endif;
return false;
}
public static function cache( $key, $value ) {
return file_put_contents( self::get_path($key), $value );
}
public static function init_buffer() {
\appcore\events::send('trace', 'creating new cache buffer');
ob_start();
}
public static function save_buffer( $key ) {
\appcore\events::send('trace', 'saving buffer to file cache');
$contents = ob_get_contents();
ob_end_clean();
self::cache( $key, $contents );
return $contents;
}
private static function get_path( $key ) {
return CACHE_PATH.md5($key).'.cache.php';
}
}
CACHE_PATH should be defined as the path to a dir, with the leading slash / and needs read+write permission.
Using it now, heres an example of how to use the file caching in a controller:
namespace app\controllers;
class index extends \appcore\base\controller {
public function index() {
$this->context->template->title = 'file caching test';
$this->context->template->show('header');
if ( \appcore\fcache::is_cached('page', 300) ):
echo \appcore\fcache::get_cache('page');
else:
\appcore\fcache::init_buffer();
echo time(), PHP_EOL;
# do some DB instensive operations here...
echo \appcore\fcache::save_buffer('page');
endif;
$this->context->template->show('footer');
}
}
Not the best use of it, but simplicity is pretty.
Check if theres a cache saved labeled "page" that is less than 300 seconds old. If found, then show the contents of that cache. Else do some intensive work, heavy db queries and save the all the outputed results to a cache named "page".
Friday, June 17, 2011
snippet test
echo "because i can";
for ($i = 0; $i < 10; $i++):
# test comment
echo "cc $i", PHP_EOL;
endfor;
Thanks to Luka Marinko and the Google hosted Code Prettify
Subscribe to:
Posts (Atom)