/dev/notes

SQL: The smallest value in a table greater than zero

A quick note to help me remember how to do this.

The problem: you want to select the smallest value from a set of values.

Let’s say you have a table of products that are in a logical group and you want to select the lowest priced product from that group, however some products actually have a 0.00 price (for whatever reason). You don’t want to actually show 0.00 as the lowest price for this group of products, you want to show the lowest price that happens to be greater than zero.

MySQL has a neat way to do this. Simply go:

SELECT tableref.group_id, MIN(NULLIF(tableref.column, 0)) as min_price FROM tableref GROUP BY tableref.group_id;

The magic is in the NULLIF function, which will return null if tableref.column is equal to 0. Returning null removes that value from inclusion by MIN, having the effect of forcing the column value to be greater than zero.

A Guide to PHP, MySQL and Nginx on Macports

By now, I’m pretty much used to and accept OSX as a desktop operating system. I remember it being quite a change when I first moved over (from Gentoo linux and Gnome 2). The mouse movement was wonky, I had to overcome years of muscle memory (learning to use the cmd instead of control key), and probably hardest of all, was leaving behind Unix’s idea of workspaces and virtual desktops. What I gave up in configurability though, was more than made up for by consistency and stability. Colleagues of mine can attest to the number of expletives launched at an emerge -vuND world that detonated my Gentoo Desktop.

So I’m happy with a less flexible, but attractive, functional and predictable desktop and I think many others feel the same way. It’s no real surprise to me then, that OSX has mostly killed off the idea of Linux on the Desktop.

But somewhere that OSX falls severely behind, is its use of a BSD inspired Unix implementation. If you’re born and raised on a diet of GNU (file|core)utils, of apt, yum, and portage, heck even sysvinit, OSX’s realisation of Unix leaves a lot to be desired.

With considerable effort and some patience though, OSX can be brought to heel. With Iterm2 and Macports you can have a functional GNUlike Unix experience.

I’ll go over the minutiae of my Macports setup another time, but generally speaking I replace all the default OSX tools with GNU equivalents and favour /opt/local/bin over everything else. It means I can have one set of configs which work mostly unchanged across Linux and OSX instances.

Macports is pretty good and the folks that contribute to it do a great job. But it does lack the polish that you take for granted with the Linux package managers. Another point to keep in mind is Macports, like Portage and BSD Ports, is a source-code based ‘package’ manager. When you install something, it is compiled right there and then on your system. When things go wrong, unless you’re a competent C programmer (and even then) you’re going to have a bad time.

One last thing to remember too, is OSX defaults to a case insensitive (but thankfully case-preserving) HFS filesystem. By default, PHP and php appear as the same thing to HFS.

So the point of this blog is to go over getting PHP running natively with Macports and how we can run an instance of Magento and the Magento Test Automation Framework (TAF).

MySQL

MySQL is probably the easiet part of the whole thing to setup. So let’s start there. For reference, the database files are stored under /opt/local/var/db/mysql55.

In Macports MySQL carrys a namespace of sorts by the way of a version suffix (as does PHP). This lets multiple versions of a package be installed side-by-side. The drawback is rather than having a mysql command, you have a mysql55 command. That’s annoying. So we will install mysql_select which lets us select a version to activate and give us proper file names.

$ sudo port install mysql55-server mysql55 mysql_select
$ sudo port select mysql mysql55
$ sudo port load mysql55-server

We will want a database for our magento application.

$ mysqladmin -uroot -p create magento 

PHP / PHP-FPM

Now we want to install PHP, PHP-FPM and the extensions Magento and TAF require.

$ sudo port install php54 php54-fpm php54-curl php54-APC php54-gd php54-pcntl php54-gd php54-mcrypt php54-iconv php54-soap php54-yaml php54-xdebug php54-openssl php54-mysql php54-pear php_select pear-PEAR

$ cd /opt/local/etc/php54
$ cp php-fpm.conf.default php-fpm.conf
$ cp php.ini-development php.ini

$ sudo vim php.ini
# set date.timezone and cgi.fix_pathinfo = 0

$ sudo vim php-fpm.conf
# make any changes for min / max num servers, error logging etc

The MySQL extension needs a little bit of prodding to look in the correct location for mysql.sock

echo 'pdo_mysql.default_socket=/opt/local/var/run/mysql55/mysqld.sock' | sudo tee --append /opt/local/var/db/mysql.ini

Once PHP-FPM is installed and configured you can use Macports to tell launchd to start it automatically.

$ sudo port load php54-fpm

PHP-Select

As with MySQL, Macports lets you install multiple versions of PHP side by side. This can be handy if you want to run PHP 5.3 and PHP 5.4 at the same time. I just install a single version, but Macports effectively namespaces everything. So rather than ‘/opt/local/bin/php’ you have ‘/opt/local/bin/php54’. PHP Select, which we installed earlier fixes this by effectively ‘activating’ one version and creating the usual executable names we’re accustomed to.

$ sudo port select php php54 

PEAR

PEAR is the single biggest pain in the whole process. And with some research it turns out its because Macports PEAR isn’t even meant be used by end users (WAT?!).

There is no MacPorts port that installs the pear package manager application with the intent that it be used by the end user outside a MacPorts port install. If you want to use pear manually on your own then you should install it using gopear, composer or some other method. http://trac.macports.org/ticket/37683

So this goes a long way to explaining why Macports doesn’t set PEAR up with sane defaults, or even put the pear command in the default path. But we can sort this all out easily enough ourselves.

$ sudo pear config-set php_bin /opt/local/bin/php
$ sudo pear config-set php_dir /opt/local/lib/php/pear
$ sudo pear config-set ext_dir /opt/local/lib/php54/extensions/no-debug-non-zts-20100525
$ sudo pear config-set bin_dir /opt/local/bin
$ sudo pear config-set cfg_dir /opt/local/lib/php/pear/cfg
$ sudo pear config-set doc_dir /opt/local/lib/php/pear/docs
$ sudo pear config-set www_dir /opt/local/lib/php/pear/www
$ sudo pear config-set test_dir /opt/local/lib/php/pear/tests
$ sudo pear config-set data_dir /opt/local/lib/php/pear/data
$ echo 'PATH=$PATH:/opt/local/lib/php/pear/bin' >> ~/.bashrc # or zshrc if you use zsh

Another issue you’ll possibly have with PEAR, is it will default to the system PHP executable (/usr/bin/php) rather than your active Macports one. The pear command does test for an environment variable so we can set up an alias to pass this variable to pear on invocation.

Add an alias to your bashrc/zshrc in the form:

alias pear='PHP_PEAR_PHP_BIN=php pear'

Reload your bashrc/zshrc.

$ source .bashrc (or source .zshrc)

Now the alias is active we can check that it’s working

$ /opt/local/lib/php/pear/bin/pear version
PEAR Version: 1.9.4
PHP Version: 5.3.15
Zend Engine Version: 2.3.0
Running on: Darwin avalanche 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

$ pear version
PEAR Version: 1.9.4
PHP Version: 5.4.12
Zend Engine Version: 2.4.0
Running on: Darwin avalanche 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

Now to make installing PEAR packages easier I turn the channel autodiscovery option on, which means you don’t have to manually add channels for package dependencies (which there are a lot when installing phing or phpunit…)

$ sudo pear config-set auto_discover 1

Now add phing and phpunit and install them with all their optional dependencies and some extra packages for the Magento TAF.

$ sudo pear channel-discover pear.phing.info
$ sudo pear channel-discover pear.phpunit.de
$ sudo pear channel-discover pear.symfony-project.com
$ sudo pear install --alldeps phing/phing 
$ sudo pear install --alldeps phpunit/phpunit
$ sudo pear install phpunit/PHP_Invoker
$ sudo pear install phpunit/PHPUnit_Selenium
$ sudo pear install -f symfony/YAML

PECL/Extensions

Macports by default creates .ini files to load extensions in /opt/local/var/db/php54. If you manually build any extensions, add the appropriate ini file here, for example:

$ echo 'extension=yaml.so' | sudo tee /opt/local/var/db/php54/yaml.ini

Nginx

Apache/Nginx. It doesn’t really matter. Both are great, but in production I use Nginx so I use it in development too. I install it with just the ssl extension enabled, to see the full range of available options, use:

$ sudo port variants nginx 

To install:

$ sudo port install nginx +ssl
$ cd /opt/local/etc/nginx
$ sudo cp fastcgi.conf.default fastcgi.conf
$ sudo cp fastcgi_params.default fastcgi_params
$ sudo cp mime.types.default mime.types
$ sudo cp nginx.conf.default nginx.conf
$ sudo mkdir conf.d sites-available sites-enabled ssl

Once installed, Nginx requires a little bit of work to hook up to PHP and particularly to work well with Magento.

$ sudo vim nginx.conf  
# Insert the following towards the bottom of the file (but inside the http block) 
map $scheme $fastcgi_https {
   default off;
   https on;
}

##
# Virtual Host Configs
##
include conf.d/*.conf;
include sites-enabled/*;

For each app just add a server block to sites-available, then symlink it to sites-enabled.

$ sudo vim sites-available/magento.dev.conf
# ...     
$ cd sites-enabled
$ sudo ln -s ../sites-available/magento.dev.conf 001-magento.dev.conf

This is the server block definition I use for magento development, feel free to modify it for your needs.

server {
    listen 80;
    listen 443 ssl;

    ssl_certificate     ssl/magento.dev.crt;
    ssl_certificate_key ssl/magento.dev.key;

    server_name magento.dev;
    root /Users/aaron/Sites/magento;

    location / {
        index index.html index.php; ## Allow a static html file to be shown first
        try_files $uri $uri/ @handler; ## If missing pass the URI to Magento's front handler
        expires 30d; ## Assume all files are cachable
    }

    ## These locations would be hidden by .htaccess normally
    location /app/                { deny all; }
    location /includes/           { deny all; }
    location /lib/                { deny all; }
    location /media/downloadable/ { deny all; }
    location /pkginfo/            { deny all; }
    location /report/config.xml   { deny all; }
    location /var/                { deny all; }
    location /shell/              { deny all; }

    ## Disable .htaccess and other hidden files
    location ~ /\. {
        deny all;
        access_log off;
        log_not_found off;
    }

    location ~ \.php$ { ## Execute PHP scripts
        if (!-e $request_filename) { rewrite / /index.php last; } ## Catch 404s that try_files miss

        expires        off; ## Do not cache dynamic content
        fastcgi_intercept_errors on;
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_param  HTTPS $fastcgi_https;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        fastcgi_param  MAGE_RUN_CODE default; ## Store code is defined in administration > Configuration > Manage Stores
        fastcgi_param  MAGE_RUN_TYPE store;
        proxy_read_timeout 120;
        proxy_connect_timeout 120;
        include        fastcgi_params; ## See /etc/nginx/fastcgi_params
    }

    location @handler { ## Magento uses a common front handler
        rewrite / /index.php;
    }
}

We’ve said our application lives on a server called ‘magento.dev’. So let’s tell our hosts file about that.

$ vim /etc/hosts
# Insert or append to an existing line
# 127.0.0.1 localhost magento.dev

Last thing that needs to be done is setting up a selfsigned ssl certificate / key pair and storing them under /opt/local/etc/nginx/ssl

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myserver.key -out myserver.crt
$ sudo mv myserver.key /opt/local/etc/nginx/etc/ssl/magento.dev.key
$ sudo mv myserver.crt /opt/local/etc/nginx/etc/ssl/magento.dev.crt

Once that’s done, we can start nginx.

$ sudo port load nginx

Web App Directory Config

I keep my web apps living under /Users/aaron/Sites, but remember that every directory element in the path needs to have the executable bit set for all users (so the web server can traverse the directory tree). Literally this is a case of:

$ chmod a+x /Users/aaron && chmod a+x /Users/aaron/Sites

Install Magento and TAF

N98 Magerun is the coolest thing to happen to Magento development since well, I can’t remember. It singlehandedly relegated a few thousand lines of cobbled together bash script to the bin.

$ cd /Users/aaron/Sites
$ curl -O magerun.phar https://github.com/netz98/n98-magerun/raw/master/n98-magerun.phar
$ chmod a+x magerun.phar
$ ./magerun.phar install
# Follow the directions and install to /Users/aaron/Sites/magento with base url http://magento.dev and database name 'magento'.

After all that work, hitting http://magento.dev should now bring up the magento demo store!

I’ve been playing with Magento’s Test Automation Framework and it was the motivation for finally getting everything working properly natively.

TAF runs at a glacial pace and in my normal development environment (VirtualBox over NFS), the universe would have undergone heat death long before the TAF suite completed its running.

Unfortunately the documentation for TAF is a bit of a mess (I’ll write about my experience with it soon), but what it offers - 1500 automated tests - is a pretty big attraction.

Installation is actually pretty easy. I am assuming you don’t have git already installed (remember you can use port variants to see what extension options are available):

$ sudo port install git-core +bash_completion +credential_osxkeychain +doc +pcre +python27
$ sudo port install git-extras
$ cd /Users/aaron/Sites
$ git clone https://github.com/magento/taf taf
$ cd taf # /Users/aaron/Sites/taf
$ cp phpunit.xml.dist phpunit.xml
$ cp config/config.yml.dist config/config.yml
$ cd .. # /Users/aaron/Sites
$ curl -O selenium-server.jar http://selenium.googlecode.com/files/selenium-server-standalone-2.31.0.jar

To run the test suite open up a new terminal

$ cd /Users/aaron/Sites
$ java -jar selenium-server.jar

Now the test suite is good to go

$ cd /Users/aaron/Sites/taf
$ ./runtests.sh

The test suite takes a loooooong time, so go for a run or something.

Hopefully these steps help out other PHP developers suffering from OSX.

MySQL with PHP and Macports

The PHP Mysql port is a little bit of a pain. By default Macports doesn’t set a default mysql sock. That leads to an error something like this:

SQLSTATE[HY000] [2002] No such file or directory.

You fix it by just appending the sock file for the mysql version you’re using into the PHP mysql.ini file. I use mysql55 so to fix PHP I do this

echo 'pdo_mysql.default_socket=/opt/local/var/run/mysql55/mysqld.sock' | sudo tee /opt/local/var/db/mysql.ini

Using SSHFS With Public Key Credentials

In the past I used to mess around with NFS over SSH but these days the FUSE options are much easier, except when you want to use a public key to authenticate with the remote host. In that case do this:

$ sshfs -o ssh_command=”ssh -i ~/ssh_keys/aaron@awshost.pem” aaron@aws.instance.com:/var/www/ ~/Sites/awshost

Actually, right after I posted this, I realised there’s a better way to do it. That is, use the ‘IdentityFile’ option instead (as per the format of .ssh/config).

$ sshfs -o "IdentityFile=~/ssh_keys/aaron@awshost.pem" aaron@aws.instance.com:/var/www/ ~/Sites/awshost

If you have any problems then add ‘-o debug’ to the above command to help track it down.

This is why people moan about PHP

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

I wanted to grab the last bit of a url that I knew would be the name of an image. I knew strstr well but that operates by giving you the remainder of a string that occurs after some needle in a string haystack. I wanted this behaviour, but only from the last instance of the needle.

strstr — Returns part of haystack string starting from and including the first occurrence of needle to the end of haystack.

strrchr -This function returns the portion of haystack which starts at the last occurrence of needle and goes until the end of haystack.

Let’s look at how these work with an example

$url = 'http://www.google.com/a/b/c/d,img';
echo strrchr($url, '/'); // prints /d.img  
echo strstr($url, '/');   // prints //www.google.com/a/b/c/d.img

Now I’ve been programming in PHP for pushing on 12 years and this one still did my head in. The names of two very similar behaving functions bare little resemblance to each other.

At this point the arguments and criticisms over the core API have been exhausted and there’s little that can/will be done. But I do wonder if it would be worth creating an object library to encapsulate primitive functions such as String, Integer, Array, Float etc., I’m not sure how possible auto-boxing is with PHP, and indeed if it’s even a good idea. But definitely some object wrappers would help ease this API pain.

Dealers of Lightning - Book Review

The best way to predict the future is to invent it. — Alan Kay

Recently I have written a little bit about Smalltalk, and in my enthusiasm I got hold of a book called Dealers of Lightning by Michael Hiltzik. It covers the rise and fall of Xerox’s Palo Alto Research Center (PARC), the research center from which Smalltalk emerged.

I initially read it to learn more about the context in which Alan Kay imagined Smalltalk and to find out who was the Executive-X he mentioned in The Early History of Smalltalk (it was Jerry Elkind). However I ended up coming away with a lot more. In particular, a new appreciation for a number of scientists I previously knew very little about. Scientists that are almost single handedly responsible for the shape of modern computing.

I grew up in the 80s so I have no real personal appreciation for computing as it was before say, 1984. I had an IBM clone (an Amstrad) and a VIC20. At school we have Commodore 64s/128s to play with. So for me, a computer has always been something that sits on your desk, you turn it on and you type away and stuff pops up on the screen. But right up until the late 70s this paradigm was considered absurd. Wasteful even. It took the vision of Alan Kay and the technical genius of Chuck Thacker and Butler Lampson along with the almost unlimited cash of Xerox to realise.

The book opens up by transporting the reader back in time to the late 60s and lays out the genesis of PARC. It then proceeds in roughly chronological order with each chapter focusing on one of the scientists and/or their inventions. The book closes by looking at some of the reasons why Xerox couldn’t transform its research into viable products. Nominally the story is about what Xerox PARC did, however Hiltzik couches everything in terms of the scientists and it is his ability to bring these characters to life that makes the book so riveting to read.

One of the most striking individuals of the story is the Impressario Robert Taylor, a man who as much as anything can be considered the grandfather of the Internet (nee ARPANET). The Kays, Thackers and Lampsons of the story are the geniuses but genius needs direction and at times support. This is the role Bob Taylor played. The story of PARC for better and worse revolves around him and his relationship with the researchers that shared his vision of interactive computing and those whether in his or the other labs, or in management, that well didn’t.

The Computer Science Lab (CSL) was a collection of engineers who weighed everything pitilessly against the question: How will this get us closer to our goal? They had commited themselves to developing Xerox’s Office of the Future and anything that diverted their attention or served an alternative goal had to be discarded or obliterated.

It is Taylor’s utterly single minded vision of interactive computing that drives much of the success and much of the drama of PARC. Taylor was in continual combat with the other labs for resources and funding and inevitably with his managers George Pake, Jerry Elkind and eventually Bill Spencer. But that was outside his lab. In it, he was the oil that kept the cogs turning and among his staff he was considered a unique and brilliant manager of researchers. It is on this skeleton of contradiction and conflict that the guts of the story of PARC hangs.

The book contains a number of particularly powerful scenes. Two particularly stuck out for me. The first is Alan Kay, his vision for a Personal Computer brusquely put down by CSL Manager Jerry Elkind, falling into a depression. Alan Kay is well known for his brilliance and verbal flourish but Hiltzik does well to also bring home his vulnerability in a way a modern reader would not expect. Kay would ultimately realise his vision of a Personal Computer - with the help of Taylor’s CSL - while Elkind was seconded away from PARC on a Xerox taskforce. We tend to recall Kay’s assertive (and largely proven) views on Computing. It is unexpected and moving then, particularly with the benefit of hindsight, to see him doubt himself and his ideas before they were fully realised.

The other scene involves Adele Goldberg, co-developer of Smalltalk, and her reactions to Apple’s infamous raid on PARC. If you’ve ever seen The Pirates of Silicon Valley you might have a feel for how this all went down. But Hiltzik’s account of it conveys such a sense of dread and hopeless frustration that the movie never came close to recreating.

By the end of the book Taylor’s time at PARC draws to a close and with his departure so too does the most storied era of PARC. Scarcely six months after Taylor’s forced resignation, the majority of his lab also resign and either follow him to Digital Equipment Corporation (creators of the famous PDP series of minicomputers), or join one of the many startups blooming in Silicon Valley following the success of IBM and Apple’s Personal Computer products.

Dealers is fundamentally a story about people that just happen to be in technology - rather than a book about technology itself. It is a human story. It is about what happens when you take the cream of a generation’s scientific talent put them in one place and throw lots of money at them. It is about what happens when you combine a visionary maverick with the proneness to credentialism by academically minded administrators. It is about what happens when you have corporate management that want to embrace change but either do not understand it, or worse, fear it.

It is the book’s focus on the people of Xerox and PARC particularly, their feelings, motivations and backgrounds that brings this extraordinary tale of modern computing’s birth to life.

Magento and GoogleCheckout Woes - Free Products

I fixed a nasty little bug in GoogleCheckout (now Wallet) today. Basically if a customer has a free or zero priced product in their cart, GoogleCheckout will return an error looking something like this:

Google Checkout: Error parsing XML; message from parser is: cvc-datatype-valid.1.2.1: ” is not a valid value for ‘decimal’.

I have developed custom modules which add free or bonus items to a customer’s cart if they use coupons, meet certain cart criteria or belong to particular customer groups. Buy x, get y rules also work this way. So this is a nuisance. Luckily few customers opt to use GoogleCheckout, but still, I don’t Live with Broken Windows[1].

Chasing the problem down the call stack leads to app/code/core/Mage/GoogleCheckout/Model/Api/Xml/Checkout.php and specifically the _getItemsXml() method.

$unitPrice = $item->getBaseCalculationPrice();
if (Mage::helper('weee')->includeInSubtotal()) {
    $unitPrice += $item->getBaseWeeeTaxAppliedAmount();
}
// ...
<unit-price currency="{$this->getCurrency()}">{$unitPrice}</unit-price>

Now, if the product’s baseprice is 0, then for some unfathomable reason it’s set to ”, not 0. As the unit-price element expects a decimal value, an empty string fails validation.

The fix is pretty trivial

$unitPrice = $item->getBaseCalculationPrice();
if (Mage::helper('weee')->includeInSubtotal()) {
    $unitPrice += $item->getBaseWeeeTaxAppliedAmount();
}

$unitPrice = ((float) $unitPrice > 0) ? $unitPrice : 0.00;

The store I needed to fix only used US dollars so I haven’t tested how the use of other currencies or locales might affect this fix.

To apply the fix, don’t modify the core codepool, but instead take advantage of the local and community codepool’s higher classloader priority[2] and place the amended code in app/code/local/Mage/GoogleCheckout/Model/Api/Xml/Checkout.php.

[1]: ‘Don’t Live With Broken Windows’ is a tip I first read about in The Pragmatic Programmer. It is used to help fight Software Entropy (software’s tendency to lose structure over time). This concept has parallels with the real world as urban areas with broken windows tend to see higher levels of vandalism when compared to areas where windows are constantly maintained.

When you ignore small problems it becomes easier to let more significant problems slide too. Hence the rule of thumb, ‘Dont Live With Broken Windows’.

[2]: Magento resolves classes in this order local, community then core. This means if two classes have the name Mage_Core_Model_Foo one exists in local the other in core, then the version in local is used.

Gracefully shutdown Nginx

When a barman calls ‘time’ at the pub, they are letting you finish your drink. Unfortunately the standard command to pull down Nginx on Ubuntu Precise is a little more aggressive. When it calls time, it snatches your unfinished beer away right there and then.

Thankfully there’s a really simple way to socialise Nginx. It is by calling the nginx server command directly with the -s argument instead of using the /etc/init.d/nginx or service nginx commands.

-s lets you send signals to the Nginx master process and Nginx behaves differently whether it receives a quit signal versus a term signal.

# terminate the nginx master process immediately
$ sudo nginx -s stop 
# terminate the nginx master process once all outstanding connections have been completed
$ sudo nginx -s quit 

$ abonner@avalanche:~$ ps aux | grep nginx
root      1063  0.0  0.0  88796  3432 ?        Ss   Jan21   0:00 nginx: master process /usr/sbin/nginx
www-data  9786  1.3  0.0  91564  7352 ?        S    Feb03 190:11 nginx: worker process is shutting down
www-data  9788  1.3  0.0  91288  7072 ?        S    Feb03 189:02 nginx: worker process is shutting down
www-data  9789  1.3  0.0  91160  6956 ?        S    Feb03 190:03 nginx: worker process is shutting down

Let you visitors finish their drink, don’t terminate nginx on a production server using /etc/init.d/nginx stop or service nginx stop.

Read more about Nginx’s command line options

(Re)Reading the Classics

Here we are already in the second week of February. I do wonder how many New Year Resolutions have survived the return to work?

This year I didn’t bother with any. I did in November though, resolve to read more technical books, and to particularly focus on the ‘classics’. My motivation was stirred by Panagiotis Louridas’s essay ‘Rereading the Classics’ from the book Beautiful Architecture. In it, Louridas examines the structure of Smalltalk and tries to reason why it achieved greater lasting success blazing a trail for others to follow than as a practical working programming environment.

Louridas suggests that Smalltalk is a classic and by way of justifying why, he quotes Italo Calvino’s Why Read the Classics (1986)

The classics […] exert a peculiar influence, both when they refuse to be eradicated from the mind and when they conceal themselves in the folds of memory, camouflaging themselves as the collective or individual unconscious.

A classic does not necessarily teach us anything we did not know before. In a classic we sometimes discover something we have always known (or thought we knew), but without knowing that this author said it first, or at least is associated with it in a special way. And this, too, is a surprise that gives much pleasure, such as we always gain from the discovery of an origin, a relationship, an affinity.

The classics are books which, upon reading, we find even fresher, more unexpected, and more marvelous than we had thought from hearing about them.

A classic is a book that comes before other classics; but anyone who has read the others first, and then reads this one, instantly recognizes its place in the family tree.

Smalltalk is a language that most of us have heard about but have rarely seen, and after being introduced to it I was inspired to start digging up any further literature I could find. The more that I read, the more that I appreciated its cleverness. Smalltalk can be eerily familiar, and as you begin to grok its syntax it is easy to recognise aspects that have inspired certain features of our ‘modern’ OO languages.

It is striking that the language itself (and indeed much of the material written about it) is so old. The most modern version of Smalltalk is Smalltalk-80 and although there are modern implementations, the overwhelming bulk of the syntax and environment still cohere to the 1980 standard. Yet download Pharo, read Alan Kay’s The Early History of Smalltalk (1993), or Dan Ingalls’s Design Principles behind Smalltalk (1981) and it all still seems contemporary. The language itself, which was controversial at the time for eschewing ALGOL syntax, has aged very well. Ruby has maybe made them fashionable, but Smalltalk sported things like block closures over a quarter of a century ago. Reflection, Metaprogramming and dynamic typing - all in Smalltalk in the 70s. Even the idea of using virtual machines to host the programming and operational environment seems remarkably contemporary today, as we increasingly move to abstract our programs and programming environments from bare metal.

It is humbling to see so many ideas that we take for granted today already implemented on a platform that is over a quarter of a century old. It is like amazing at how the Egyptians built the pyramids. Sure we could do it now, no sweat. But we have so much more raw engineering knowledge to throw at the problem. Alan Kay, Dan Ingalls, Adele Goldberg and the rest of the team at the Xerox PARC Learning Research Group designed and implemented Smalltalk with less computing power than my fridge has today. Despite formalising most of the vocabulary for OO software development during the development of Smalltalk it’s hard not to feel like some of the energy and innovation of Kay’s thinking didn’t survive the C++ and Java succession.

Our discipline is still quite young when compared to traditional engineering or the sciences. Yet we seem to keep facing the same problems over and over again. The only difference is perhaps a few orders of abstraction, bigger piles of data and slightly more exotic technologies. But when I think that the fundamental concept of ‘Agile’ or at least ‘iterative’ development was doing the rounds in the 60s, it made me wonder what other insights are out there, buried in the forgotten past.

Maybe one reason we tend to forget what others have learned is that the average developer only reads one technical book per year. That means a sizeable percentage of software professionals do not actually read any! I think if I average over my professional working career (after graduating at the end of 2004) I would probably be batting one per year too. If I were to look over the last three years, maybe two per year. I wonder how many mistakes I might have avoided in that period had I read Fred Brooks’s Mythical Man Month in 2004 rather than 2012.

My excuse has always been a lack of quality time. Last year in July I left my fulltime job for the world of consulting/freelancing. I imagined it would be easier to read more. In practice it has been, but not as easy as I thought or hoped.

I have a young daughter, my partner has returned to work and despite working from home, I still put in 50-60 hour weeks. It doesn’t leave a lot of spare time and what spare time there is, is usually late at night when it’s hard to concentrate.

So since I made a determined effort to read more, I’ve read four books cover to cover, and cherry picked bits out of another four. It feels good and the trick I’ve found is to read a chapter at a time, whenever you can. Whether it’s just before dinner, over coffee/lunch, waiting at the supermarket or just before bed. I found that by reading, even if was just a little bit, every day, I was starting to get through entire books.

It requires conscious effort though, and some material is more suited to this style of reading than others. After a long day just before bed it’s pointless trying to delve into SICP. It helps to read certain books during the work day, for example I’ve been re-reading GOOS and going through Kent Beck’s TDD by Example over lunch times. I actively practice TDD so reading a chapter from one of these books midday helps me relate it directly to what I’m working on when I go back into the office.

I have a huge reading list setup in Google Reader, and I’m starting to think it’s distracting. I have a slight OCD in that I need to keep the unread count at zero. At the end of a thirty minute work sprint, I would take five minutes to quickly flick through the list. It is distracting actually, and most of the content is superficial. The benefit of reading a book over a blog is the tendancy for a book to have its ideas more fully formed and logically structured. I realise the irony saying this while writing a blog myself. I feel that blogs have their place, but I have been spending far more time reading blogs than reading books. I am now leaving my Google Reader list unread longer and after a few 30 minute sprints picking up a book instead.

As I wrote above, Louridas inspired me to learn more about Smalltalk, and any research into the language leads to Alan Kay’s ACM paper The Early History of Smalltalk. The paper beyond providing a wonderful insight into the language and the Xerox PARC is the source of some great quotes. One of my favourites goes:

Where Newton said he saw further by standing on the shoulders of giants, computer scientists all too often stand on each other’s toes.

In Software Engineering, we are so busy looking forward that we don’t look back often enough. There is such a rich wealth of knowledge out there already considered and documented. I think we all should make more of an effort to re-discover it.

Some Hidden Unix / Shell Gems

It is 2013, we (still) don’t have flying cars, or hoverboards, AND as developers, we still use terminals to interact with our operating system. So every so often I like to browse through commandlinefu.com and try and pick up any little tidbits which improve my command line efficiency.

Here’s a small selection I have picked up recently that I didn’t know.

sudo !! 

Run the previous command as sudo. This is great when you realise you needed to run something as root.

ctrl-x e

Open up $EDITOR to enter in a long command. In my setup it fires up vim. This is great for some of the long rails commands you need to create controllers or models.

cat /etc/passwd | column -s':' -t 

Column, columnates input, the -t argument will format standard input into a table and -s lets you specify an arbitrary field delimiter. For unformatted input this is very handy.

These next few are specific to zsh, and while I do love bash, since switching to zsh I haven’t really looked back. It’s things like this that when you work with a terminal every single day, you can’t give up.

aaron@tempest ~ $ d                                 
0   ~
aaron@tempest ~ $ cd /etc
aaron@tempest /etc $ d
0   /etc
1   ~
aaron@tempest /etc $ 1
~
aaron@tempest ~ $  

The ‘d’ command lists the directory stack, and then entering an integer will switch you directly to the directory index in the stack. It is a killer app.

Moving directories also is very pleasant in zsh. Use ‘..’ to move up a directory, and simply type the name of the directory in, to move into a directory.

aaron@tempest ~ $ ..       
aaron@tempest /Users $ aaron
aaron@tempest ~ $ 

This last one is a trick I’ve know for a few years, I don’t know how much time this has saved me exactly, but I use it every single day.

In vim, if you’re editing a file that requires root (or any other user) permissions, you can write the file by doing

:w !sudo tee %

I use it so much that I’ve set up a leader key binding in my .vimrc

nnoremap <leader>sr :w !sudo tee %<CR>

There’s nothing more annoying than making lengthy changes to a config file, go to write it and getting permission denied…

I make all my configs available online at github, if you’re interested in seeing how I setup my environment.

Building the Mysql2 Gem with Macports

Macports puts its libraries in non-standard locations, so to build the mysql2 gem on an OSX computer, you will need to do a little bit of extra work to ensure that gem calls make with appropriate options.

To cut a short story, very short, you do this (at least if you have macports in /opt/local (the default), and are using the mysql55 package).

$ gem install mysql2 -- --with-mysql-lib=/opt/local/lib/mysql55/mysql --with-mysql-include=/opt/local/include/mysql55/mysql

Remote CLI XDebugging with PHPUnit

There’s a peculiar issue right now with PHPUnit where it will not respect php.ini arguments supplied to it on the commandline (i.e. supplying -d arguments).

This matters a lot when you want to use xdebug on a project that runs off a virtual machine, or even perhaps a remote server.

The typical pattern (when using PHPStorm in my case) to invoke a remote cli debugging session is to set an environment variable telling the IDE what server configuration to use, and to tell PHP what remote host to connect to.

$ PHP_IDE_CONFIG='serverName=mydevmachine.local' php -dxdebug.remote_host=192.168.0.1 myphpscript.php

Now this will work fine, however if we want to debug during a phpunit test normally you would do this

$ PHP_IDE_CONFIG='serverName=mydevmachine.local' phpunit -dxdebug.remote_host=192.168.0.1 -c phpunit.xml

Unfortunately this doesn’t appear to work at the moment (version 3.7.9). If I use the xdebug test client, I can see xdebug trying to connect to the localhost, ignoring what I’ve told PHPUnit. I’ll look into this a bit more later, but I suspect PHPUnit isn’t passing on the php.ini settings in a timely fashion for xdebug to hook into.

The solution to this problem is to make use of ssh port forwarding. This works exactly the same for a virtual machine as it would for a remote host, which makes xdebugging on a production machine (hopefully only ever in an emergency!!!) much more straight forward (and less insecure).

$ ssh -R 9000:localhost:9000 myvm.local

This sets up myvm.local to forward all connections to its localhost on port 9000 to the remote client’s port 9000. When xdebug goes to connect to localhost:9000, it ends up actually connecting to mydevmachine.local:9000.

It’s a bit of a hack, but a time saving one. The other alternative is Vim and its xdebug plugin. This isn’t a bad alternative. But once you’ve experienced the power of PHPStorm’s debugging implementation it’s hard to go back.

A Great Time to be a PHP Developer

Embrace change

It has been an exciting time to be a PHP Developer these past twelve months, PHP 5.3 is now rock solid and PHP 5.4 is getting there. Both releases significantly modernise elements of the language, closing the gap between PHP and the offerings of more ‘in vogue’ languages.

In technology we often see change happen in sudden, explosive steps. Often it seems to coincide with developments a technology’s ecosystem or among its competitors. For PHP the first major kick was the rapid rise in popularity of Object Oriented Programming in the early 00s. This led to PHP 5’s radically overhauled OO implementation in 2004. The next kick, I feel, came in 2005 when Ruby on Rails exploded into everyone’s consciousness. RoR provided a full stack web development platform that drastically simplified creating complex web applications. The PHP community responded in kind with a number of ‘fullfat’ Model View Controller (MVC) Frameworks, the most successful being Zend’s and Symfony.

It all depends

The arrival of PHP 5.3 and features like Namespaces, PHAR, Closures and the ubiquity of Github is having the effect of giving PHP a new kick, and the results are starting to make themselves felt. We now have second generation frameworks from Zend and Symfony leveraging these technologies.

One problem remains though, and that is managing and distributing dependencies. Modern web development platforms all now have robust dependency management tools available and in the PHP camp, PEAR wasn’t really cutting it.

The success of Symfony2 in particular, with its emphasis on high quality, modular components, forced PHP developers to address how they bundled and distributed library code.

Luckily for us, the guys behind Composer, (again, taking considerable cues from the Ruby community) have licked it. Composer, in tandem with Symfony2 components allow PHP Developers to confidently build on top of other developers’ libraries.

Do we really need another packaging tool?

Why did we need another package and dependency management tool anyway? What really, is wrong with PEAR? Well, if we wind the clock way back to 1999 when Netscape Communicator was still the most popular web browser and Google had just moved out of Susan Wojcicki’s garage, PEAR was conceived as PHP’s answer to PERL’s CPAN. Despite some strident efforts, it never really managed to become the most pleasant package manager to work with: rigid, elitist and worst of all, difficult for end-users. PEAR’s age strictly speaking is not the problem, but its centralised nature is a bottleneck and there is no straightforward way to handle two packages with varying dependencies. For example: say package x, requires stable package y. Package z requires beta package y. You can’t install both. Dependency and package management has moved on a long way since 1999.

Over time PEAR’s shortcomings have led to a graveyard of abandoned packages, code of at best variable and at worst, dubious, quality, and a community lacking in any sort of dynamism. If you make something easy, people will use it. PEAR is difficult to use for developers and users alike.

Composer democratises (in the best sense) things and puts full control of dependencies in the hands of library developers. Free to pick and chose code they want to use. Free from having to worry about navigating the PEAR jungle. Here the rise and rise of Github has been key. Composer can sit over the top of code distribution services provided by Github, or it can use its default Packagist repository. This removes the need for libraries to live in a blessed canonical repository or for developers to host it themselves.

… profit?

There’s no compelling need now to constantly rewrite basic library components (I think we’ve finally licked what ought to be the basic issue of class loading!), Free of the shackles of PEAR, we are witnessing an explosion of high quality PHP frameworks, libraries and utilities.

PHPSpec, Behat, Twig, Mockery, Doctrine, are just a few that immediately spring to mind. Some (such as Doctrine) have been around a while. However the advances PHP 5.3 brought to the table have significantly helped improve the utility of these projects.

Anyway, so (after a fashion) I come to the tool that motivated me to write this post, n98-magerun.

The name is horrible, but the tool itself is brilliant. In short, it’s Drush for Magento and it’s wonderful. It is one of those tools that makes you wonder what on earth you did before it.

I have a folder full of bash scripts, cobbled together to help automate the mind-numbing process of managing Magento installations. Over the course of a few months Christian Münch and friends have overseen a small tool quickly develop into the kind of utility we’ve all wanted but never had the time/patience to build ourselves.

Magerun is elegantly simple for the user, and cleanly extendable by developers. It is a perfect illustration of why it’s such a great time to be a PHP developer. Better dependency management, easy distribution, modular libraries and powerful language syntax have all came together to let someone with an itch, scratch it quickly and effectively.

It has become several orders of magnitude easier to develop, package and distribute PHP libraries and utilities. The result of this leap forward is a brilliant tool that helps Magento developers dramatically increase their productivity.

MageRun and Xmllint: Pretty Printing Magento XML Config

Just a quick note, as you may notice from the comments, Magerun now pretty prints the xml output by default. It appears DomDocument requires preserveWhitespace = false in order to correctly reformat output. Thanks to Christian for sorting it all out!

I’ll be writing about how awesome Magerun is shortly, but just one of its cool features is the ability to dump out a merged version of Magento’s config.

This is extremely helpful when trying to resolve conflicts between modules, or figure out what bit of configuration is taking precedence.

The resulting xml though is pretty raw and unformatted, but xmllint can fix that.

Xmllint expects a file to work with and cannot use piped input. So we need to use bash’s Process Substitution feature to avoid having to create temporary files.

$ xmllint --format <(magerun config:dump)

So, magerun and xmllint, a simple way to get a formatted, easy to examine view of how Magento is putting your install’s configuration together.