I use it as part of my git workflow (see my magento githooks repo) and I find it really does help keep code consistent.

When you work on projects with more than a couple of developers, niggling differences in style can lead to unintentional errors creeping into your codebase (think omitting braces in if statements, for example).

Anyway, one of the neat features in the master branch is the ability to specify a config file to control how php-cs-fixer behaves rather than having to define everything on the commandline (or depend on .php_cs being in the path). Unfortunately this feature is not yet in a pre-baked release of php-cs-fixer. So to use it you have to use the source version. I quite like the convenience of the phar version but it's unclear how to build one directly from the sources.

After a bit of digging around in the code and issues list, I found they are using the php-box project to build release phars. It's actually very simple, but to save others having to figure it all out, just follow these steps.

$ git clone https://github.com/fabpot/PHP-CS-Fixer
$ cd PHP-CS-Fixer
$ composer.phar require --dev 'kherge/box=~2.4'
$ vendor/kherge/box/bin/box build
> Building...

Do a quick ls and you'll notice you have a minty fresh php-cs-fixer.phar file.

Done.

$ usermod -a -G thegroup theuser

And job done. But OSX uses Open Directory rather than traditional flatfiles like /etc/passwd and /etc/group to store information about users and domains. So the typical unix commands we are used to, don't work.

The dscl (directory service command line) utility lets you manipulate Open Directory values and in our case, add a user to an additional group, handy if for example, you want to add your user to the wheel group to make use of password free sudo.

$ dscl localhost --append /Local/Default/Groups/<groupnamehere> <usernamehere>

So, later on, you'll probably want to pull changes down from the remote and you'll end up seeing something similar to this

➜  store git:(zendesk) git pull --rebase                                                          [48/53]There is no tracking information for the current branch.
Please specify which branch you want to rebase against.
See git-pull(1) for details

git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

git branch --set-upstream-to=origin/<branch> zendesk

Now it's not that hard to type out the suggested command above to set the upstream branch, but I got sick of having to do it so often, and I have given up trying to remember -u, so I created a git alias to automate things and save some keystrokes.

In your ~/.gitconfig under the alias section, add this

    sup = !git branch --set-upstream-to=origin/`git symbolic-ref --short HEAD`

You can use the alias by issuing the following command in your terminal

$ git sup

This will look at the current branch and set its upstream to origin/branchname

If you tend to use another remote name other than origin, change the alias accordingly.

I have a few other useful aliases which you can checkout (hah, sorry :)) in my full gitconfig.

For some workarounds checkout http://mywiki.wooledge.org/BashFAQ/024

Usually these types of commits can be avoided by keeping feature branches up to date with git --rebase. When two branches have a direct common history, merges can be applied using the fast-forward strategy avoiding the need for a stitch-things-together merge commit.

Because the commit pointed to by the branch you merged in was directly upstream of the commit you’re on, Git moves the pointer forward. To phrase that another way, when you try to merge one commit with a commit that can be reached by following the first commit’s history.

http://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging

To ensure you've kept your branches synced up with rebase and to avoid accidentally creating a merge commit, you can set git merge to only perform fast forward merges.

$ git config --global merge.ff only

This way, you'll get a gentle reminder to rebase. If that's not feasible then you can force through the merge with

$ git merge --no-ff

• grumpy-phpunit.com
• Price: $29.00

The Grumpy Programmer (actual name Chris Hartjes) amusingly blogs and tweets all things PHP and particularly PHPUnit. When I saw he was publishing this book, I was curious to see how his strident style would stand up to the longer form. Pleasantly, it turns out.

Chris maintains his gruff voice while whirling through the ins and outs of using PHPUnit. I've been using PHPUnit for a long time now and I find that when I am really familiar with a tool, I tend to re/(over)use patterns that have served me well in the past. So while I found I was familiar with much of the material in the book, there are more than a few tidbits here that I picked up. I feel even the most grizzled PHPUnit veteran's testing regime will benefit from a read through.

The book seems aimed at the less experienced which I did find a little surprising given the title. When I think cookbook I tend to think of the weighty O'Reilly tomes. This book though is more like a lengthy tutorial than a cookbook in the O'Reilly style. As a tutorial goes though it excels. It is detailed without being turgid and covers all the major aspects of using PHPUnit that I would expect it to and then some. I found the chapter on Test Doubles (that is mocks, stubs and fakes) to be particularly excellent. The vocabulary surrounding these terms tends to get mixed up and consequently programmers often treat them as the same thing. That leads in my experience, at best, to confusion. And at worst, to poor tests that are difficult to maintain.

As a quick aside, the book is published by LeanPub, who ensure authors receive 90% of the proceeds from their work. I think this is a wonderful initiative. Writing, especially for a programmer is tremendously hard and I like the idea of those that attempt it, and do a good job, get appropriately rewarded for doing so.

So, back to the book. You find peppered throughout the introduction to PHPUnit subtle wisdoms that are hard to argue with. A simple and you would think obvious example, is that of always providing the final argument to assert statements with a description message. This message is displayed when the test fails, helping you quickly identify where the problem lies. Another: writing strictly encapsulated code that eschews static methods and class variables, is (well unsurprisingly) easier to test that code that is constantly mutating global state.

The book is quite short, coming in (at least in my pdf version) at 85 pages. I feel like there is sufficient scope for more content here. Especially for a 'cookbook'. I would have loved to have seen more on using Data Builders for example. The chapter on data providers is great, but I find you often need more fine grained control over your fixtures. Factories and data builders are a couple concepts that once learned, significantly reduce the friction of TDD.

I perhaps would also liked to have seen more in introduction to TDD itself, motivations for it, and perhaps a brief comparison between the two principle TDD xUnit styles. Specifically the Statist TDD and Mockist/London School TDD styles. The former being a test style mainly interested in setting up some state, running a behaviour and checking the end state matches what you expected. The Mockist approach is less interested in observing State and instead is more interested in the messages passed between objects (method calls between collaborators).

Overall I enjoyed the book, and it fills a much needed role in guiding budding PHP TDD practitioners in the use of the most mature tool we have available in PHP. I picked up a few neat new tricks and I suspect many PHP programmers will do the same.

You can buy it now at grumpy-phpunit.com

The problem: you want to select the smallest value from a set of values.

Let's say you have a table of products that are in a logical group and you want to select the lowest priced product from that group, however some products actually have a 0.00 price (for whatever reason). You don't want to actually show 0.00 as the lowest price for this group of products, you want to show the lowest price that happens to be greater than zero.

MySQL has a neat way to do this. Simply go:

SELECT tableref.group_id, MIN(NULLIF(tableref.column, 0)) as min_price FROM tableref GROUP BY tableref.group_id;

The magic is in the NULLIF function, which will return null if tableref.column is equal to 0. Returning null removes that value from inclusion by MIN, having the effect of forcing the column value to be greater than zero.

So I'm happy with a less flexible, but attractive, functional and predictable desktop and I think many others feel the same way. It's no real surprise to me then, that OSX has mostly killed off the idea of Linux on the Desktop.

But somewhere that OSX falls severely behind, is its use of a BSD inspired Unix implementation. If you're born and raised on a diet of GNU (file|core)utils, of apt, yum, and portage, heck even sysvinit, OSX's realisation of Unix leaves a lot to be desired.

With considerable effort and some patience though, OSX can be brought to heel. With Iterm2 and Macports you can have a functional GNUlike Unix experience.

I'll go over the minutiae of my Macports setup another time, but generally speaking I replace all the default OSX tools with GNU equivalents and favour /opt/local/bin over everything else. It means I can have one set of configs which work mostly unchanged across Linux and OSX instances.

Macports is pretty good and the folks that contribute to it do a great job. But it does lack the polish that you take for granted with the Linux package managers. Another point to keep in mind is Macports, like Portage and BSD Ports, is a source-code based 'package' manager. When you install something, it is compiled right there and then on your system. When things go wrong, unless you're a competent C programmer (and even then) you're going to have a bad time.

One last thing to remember too, is OSX defaults to a case insensitive (but thankfully case-preserving) HFS filesystem. By default, PHP and php appear as the same thing to HFS.

So the point of this blog is to go over getting PHP running natively with Macports and how we can run an instance of Magento and the Magento Test Automation Framework (TAF).

MySQL

MySQL is probably the easiet part of the whole thing to setup. So let's start there. For reference, the database files are stored under /opt/local/var/db/mysql55.

In Macports MySQL carrys a namespace of sorts by the way of a version suffix (as does PHP). This lets multiple versions of a package be installed side-by-side. The drawback is rather than having a mysql command, you have a mysql55 command. That's annoying. So we will install mysql_select which lets us select a version to activate and give us proper file names.

$ sudo port install mysql55-server mysql55 mysql_select
$ sudo port select mysql mysql55
$ sudo port load mysql55-server

We will want a database for our magento application.

$ mysqladmin -uroot -p create magento 

PHP / PHP-FPM

Now we want to install PHP, PHP-FPM and the extensions Magento and TAF require.

$ sudo port install php54 php54-fpm php54-curl php54-APC php54-gd php54-pcntl php54-gd php54-mcrypt php54-iconv php54-soap php54-yaml php54-xdebug php54-openssl php54-mysql php54-pear php_select pear-PEAR

$ cd /opt/local/etc/php54
$ cp php-fpm.conf.default php-fpm.conf
$ cp php.ini-development php.ini

$ sudo vim php.ini
# set date.timezone and cgi.fix_pathinfo = 0

$ sudo vim php-fpm.conf
# make any changes for min / max num servers, error logging etc

The MySQL extension needs a little bit of prodding to look in the correct location for mysql.sock

echo 'pdo_mysql.default_socket=/opt/local/var/run/mysql55/mysqld.sock' | sudo tee --append /opt/local/var/db/mysql.ini

Once PHP-FPM is installed and configured you can use Macports to tell launchd to start it automatically.

$ sudo port load php54-fpm

PHP-Select

As with MySQL, Macports lets you install multiple versions of PHP side by side. This can be handy if you want to run PHP 5.3 and PHP 5.4 at the same time. I just install a single version, but Macports effectively namespaces everything. So rather than '/opt/local/bin/php' you have '/opt/local/bin/php54'. PHP Select, which we installed earlier fixes this by effectively 'activating' one version and creating the usual executable names we're accustomed to.

$ sudo port select php php54 

PEAR

PEAR is the single biggest pain in the whole process. And with some research it turns out its because Macports PEAR isn't even meant be used by end users (WAT?!).

There is no MacPorts port that installs the pear package manager application with the intent that it be used by the end user outside a MacPorts port install. If you want to use pear manually on your own then you should install it using gopear, composer or some other method. http://trac.macports.org/ticket/37683

So this goes a long way to explaining why Macports doesn't set PEAR up with sane defaults, or even put the pear command in the default path. But we can sort this all out easily enough ourselves.

$ sudo pear config-set php_bin /opt/local/bin/php
$ sudo pear config-set php_dir /opt/local/lib/php/pear
$ sudo pear config-set ext_dir /opt/local/lib/php54/extensions/no-debug-non-zts-20100525
$ sudo pear config-set bin_dir /opt/local/bin
$ sudo pear config-set cfg_dir /opt/local/lib/php/pear/cfg
$ sudo pear config-set doc_dir /opt/local/lib/php/pear/docs
$ sudo pear config-set www_dir /opt/local/lib/php/pear/www
$ sudo pear config-set test_dir /opt/local/lib/php/pear/tests
$ sudo pear config-set data_dir /opt/local/lib/php/pear/data
$ echo 'PATH=$PATH:/opt/local/lib/php/pear/bin' >> ~/.bashrc # or zshrc if you use zsh

Another issue you'll possibly have with PEAR, is it will default to the system PHP executable (/usr/bin/php) rather than your active Macports one. The pear command does test for an environment variable so we can set up an alias to pass this variable to pear on invocation.

Add an alias to your bashrc/zshrc in the form:

alias pear='PHP_PEAR_PHP_BIN=php pear'

Reload your bashrc/zshrc.

$ source .bashrc (or source .zshrc)

Now the alias is active we can check that it's working

$ /opt/local/lib/php/pear/bin/pear version
PEAR Version: 1.9.4
PHP Version: 5.3.15
Zend Engine Version: 2.3.0
Running on: Darwin avalanche 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

$ pear version
PEAR Version: 1.9.4
PHP Version: 5.4.12
Zend Engine Version: 2.4.0
Running on: Darwin avalanche 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

Now to make installing PEAR packages easier I turn the channel autodiscovery option on, which means you don't have to manually add channels for package dependencies (which there are a lot when installing phing or phpunit…)

$ sudo pear config-set auto_discover 1

Now add phing and phpunit and install them with all their optional dependencies and some extra packages for the Magento TAF.

$ sudo pear channel-discover pear.phing.info
$ sudo pear channel-discover pear.phpunit.de
$ sudo pear channel-discover pear.symfony-project.com
$ sudo pear install --alldeps phing/phing 
$ sudo pear install --alldeps phpunit/phpunit
$ sudo pear install phpunit/PHP_Invoker
$ sudo pear install phpunit/PHPUnit_Selenium
$ sudo pear install -f symfony/YAML

PECL/Extensions

Macports by default creates .ini files to load extensions in /opt/local/var/db/php54. If you manually build any extensions, add the appropriate ini file here, for example:

$ echo 'extension=yaml.so' | sudo tee /opt/local/var/db/php54/yaml.ini

Nginx

Apache/Nginx. It doesn't really matter. Both are great, but in production I use Nginx so I use it in development too. I install it with just the ssl extension enabled, to see the full range of available options, use:

$ sudo port variants nginx 

To install:

$ sudo port install nginx +ssl
$ cd /opt/local/etc/nginx
$ sudo cp fastcgi.conf.default fastcgi.conf
$ sudo cp fastcgi_params.default fastcgi_params
$ sudo cp mime.types.default mime.types
$ sudo cp nginx.conf.default nginx.conf
$ sudo mkdir conf.d sites-available sites-enabled ssl

Once installed, Nginx requires a little bit of work to hook up to PHP and particularly to work well with Magento.

$ sudo vim nginx.conf  
# Insert the following towards the bottom of the file (but inside the http block) 
map $scheme $fastcgi_https {
   default off;
   https on;
}

##
# Virtual Host Configs
##
include conf.d/*.conf;
include sites-enabled/*;

For each app just add a server block to sites-available, then symlink it to sites-enabled.

$ sudo vim sites-available/magento.dev.conf
# ...     
$ cd sites-enabled
$ sudo ln -s ../sites-available/magento.dev.conf 001-magento.dev.conf

This is the server block definition I use for magento development, feel free to modify it for your needs.

server {
    listen 80;
    listen 443 ssl;
    
    ssl_certificate     ssl/magento.dev.crt;
    ssl_certificate_key ssl/magento.dev.key;

    server_name magento.dev;
    root /Users/aaron/Sites/magento;

    location / {
        index index.html index.php; ## Allow a static html file to be shown first
        try_files $uri $uri/ @handler; ## If missing pass the URI to Magento's front handler
        expires 30d; ## Assume all files are cachable
    }

    ## These locations would be hidden by .htaccess normally
    location /app/                { deny all; }
    location /includes/           { deny all; }
    location /lib/                { deny all; }
    location /media/downloadable/ { deny all; }
    location /pkginfo/            { deny all; }
    location /report/config.xml   { deny all; }
    location /var/                { deny all; }
    location /shell/              { deny all; }

    ## Disable .htaccess and other hidden files
    location ~ /\. {
        deny all;
        access_log off;
        log_not_found off;
    }

    location ~ \.php$ { ## Execute PHP scripts
        if (!-e $request_filename) { rewrite / /index.php last; } ## Catch 404s that try_files miss

        expires        off; ## Do not cache dynamic content
        fastcgi_intercept_errors on;
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_param  HTTPS $fastcgi_https;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        fastcgi_param  MAGE_RUN_CODE default; ## Store code is defined in administration > Configuration > Manage Stores
        fastcgi_param  MAGE_RUN_TYPE store;
        proxy_read_timeout 120;
        proxy_connect_timeout 120;
        include        fastcgi_params; ## See /etc/nginx/fastcgi_params
    }
    
    location @handler { ## Magento uses a common front handler
        rewrite / /index.php;
    }
}

We've said our application lives on a server called 'magento.dev'. So let's tell our hosts file about that.

$ vim /etc/hosts
# Insert or append to an existing line
# 127.0.0.1 localhost magento.dev

Last thing that needs to be done is setting up a selfsigned ssl certificate / key pair and storing them under /opt/local/etc/nginx/ssl

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myserver.key -out myserver.crt
$ sudo mv myserver.key /opt/local/etc/nginx/etc/ssl/magento.dev.key
$ sudo mv myserver.crt /opt/local/etc/nginx/etc/ssl/magento.dev.crt

Once that's done, we can start nginx.

$ sudo port load nginx

Web App Directory Config

I keep my web apps living under /Users/aaron/Sites, but remember that every directory element in the path needs to have the executable bit set for all users (so the web server can traverse the directory tree). Literally this is a case of:

$ chmod a+x /Users/aaron && chmod a+x /Users/aaron/Sites

Install Magento and TAF

N98 Magerun is the coolest thing to happen to Magento development since well, I can't remember. It singlehandedly relegated a few thousand lines of cobbled together bash script to the bin.

$ cd /Users/aaron/Sites
$ curl -O magerun.phar https://github.com/netz98/n98-magerun/raw/master/n98-magerun.phar
$ chmod a+x magerun.phar
$ ./magerun.phar install
# Follow the directions and install to /Users/aaron/Sites/magento with base url http://magento.dev and database name 'magento'.

After all that work, hitting http://magento.dev should now bring up the magento demo store!

I've been playing with Magento's Test Automation Framework and it was the motivation for finally getting everything working properly natively.

TAF runs at a glacial pace and in my normal development environment (VirtualBox over NFS), the universe would have undergone heat death long before the TAF suite completed its running.

Unfortunately the documentation for TAF is a bit of a mess (I'll write about my experience with it soon), but what it offers - 1500 automated tests - is a pretty big attraction.

Installation is actually pretty easy. I am assuming you don't have git already installed (remember you can use port variants to see what extension options are available):

$ sudo port install git-core +bash_completion +credential_osxkeychain +doc +pcre +python27
$ sudo port install git-extras
$ cd /Users/aaron/Sites
$ git clone https://github.com/magento/taf taf
$ cd taf # /Users/aaron/Sites/taf
$ cp phpunit.xml.dist phpunit.xml
$ cp config/config.yml.dist config/config.yml
$ cd .. # /Users/aaron/Sites
$ curl -O selenium-server.jar http://selenium.googlecode.com/files/selenium-server-standalone-2.31.0.jar

To run the test suite open up a new terminal

$ cd /Users/aaron/Sites
$ java -jar selenium-server.jar

Now the test suite is good to go

$ cd /Users/aaron/Sites/taf
$ ./runtests.sh

The test suite takes a loooooong time, so go for a run or something.

Hopefully these steps help out other PHP developers suffering from OSX.

SQLSTATE[HY000] [2002] No such file or directory.

You fix it by just appending the sock file for the mysql version you're using into the PHP mysql.ini file. I use mysql55 so to fix PHP I do this

echo 'pdo_mysql.default_socket=/opt/local/var/run/mysql55/mysqld.sock' | sudo tee /opt/local/var/db/mysql.ini

$ sshfs -o ssh_command="ssh -i ~/ssh_keys/aaron@awshost.pem" aaron@aws.instance.com:/var/www/ ~/Sites/awshost

Actually, right after I posted this, I realised there's a better way to do it. That is, use the 'IdentityFile' option instead (as per the format of .ssh/config).

$ sshfs -o "IdentityFile=~/ssh_keys/aaron@awshost.pem" aaron@aws.instance.com:/var/www/ ~/Sites/awshost

If you have any problems then add '-o debug' to the above command to help track it down.

There are only two hard things in Computer Science: cache invalidation and naming things.

-- Phil Karlton

I wanted to grab the last bit of a url that I knew would be the name of an image. I knew strstr well but that operates by giving you the remainder of a string that occurs after some needle in a string haystack. I wanted this behaviour, but only from the last instance of the needle.

strstr — Returns part of haystack string starting from and including the first occurrence of needle to the end of haystack.

strrchr -This function returns the portion of haystack which starts at the last occurrence of needle and goes until the end of haystack.

Let's look at how these work with an example

$url = 'http://www.google.com/a/b/c/d,img';
echo strrchr($url, '/'); // prints /d.img  
echo strstr($url, '/');   // prints //www.google.com/a/b/c/d.img

Now I've been programming in PHP for pushing on 12 years and this one still did my head in. The names of two very similar behaving functions bare little resemblance to each other.

At this point the arguments and criticisms over the core API have been exhausted and there's little that can/will be done. But I do wonder if it would be worth creating an object library to encapsulate primitive functions such as String, Integer, Array, Float etc., I'm not sure how possible auto-boxing is with PHP, and indeed if it's even a good idea. But definitely some object wrappers would help ease this API pain.

The best way to predict the future is to invent it. -- Alan Kay

Recently I have written a little bit about Smalltalk, and in my enthusiasm I got hold of a book called Dealers of Lightning by Michael Hiltzik. It covers the rise and fall of Xerox's Palo Alto Research Center (PARC), the research center from which Smalltalk emerged.

I initially read it to learn more about the context in which Alan Kay imagined Smalltalk and to find out who was the Executive-X he mentioned in The Early History of Smalltalk (it was Jerry Elkind). However I ended up coming away with a lot more. In particular, a new appreciation for a number of scientists I previously knew very little about. Scientists that are almost single handedly responsible for the shape of modern computing.

I grew up in the 80s so I have no real personal appreciation for computing as it was before say, 1984. I had an IBM clone (an Amstrad) and a VIC20. At school we have Commodore 64s/128s to play with. So for me, a computer has always been something that sits on your desk, you turn it on and you type away and stuff pops up on the screen. But right up until the late 70s this paradigm was considered absurd. Wasteful even. It took the vision of Alan Kay and the technical genius of Chuck Thacker and Butler Lampson along with the almost unlimited cash of Xerox to realise.

The book opens up by transporting the reader back in time to the late 60s and lays out the genesis of PARC. It then proceeds in roughly chronological order with each chapter focusing on one of the scientists and/or their inventions. The book closes by looking at some of the reasons why Xerox couldn't transform its research into viable products. Nominally the story is about what Xerox PARC did, however Hiltzik couches everything in terms of the scientists and it is his ability to bring these characters to life that makes the book so riveting to read.

One of the most striking individuals of the story is the Impressario Robert Taylor, a man who as much as anything can be considered the grandfather of the Internet (nee ARPANET). The Kays, Thackers and Lampsons of the story are the geniuses but genius needs direction and at times support. This is the role Bob Taylor played. The story of PARC for better and worse revolves around him and his relationship with the researchers that shared his vision of interactive computing and those whether in his or the other labs, or in management, that well didn't.

The Computer Science Lab (CSL) was a collection of engineers who weighed everything pitilessly against the question: How will this get us closer to our goal? They had commited themselves to developing Xerox's Office of the Future and anything that diverted their attention or served an alternative goal had to be discarded or obliterated.

It is Taylor's utterly single minded vision of interactive computing that drives much of the success and much of the drama of PARC. Taylor was in continual combat with the other labs for resources and funding and inevitably with his managers George Pake, Jerry Elkind and eventually Bill Spencer. But that was outside his lab. In it, he was the oil that kept the cogs turning and among his staff he was considered a unique and brilliant manager of researchers. It is on this skeleton of contradiction and conflict that the guts of the story of PARC hangs.

The book contains a number of particularly powerful scenes. Two particularly stuck out for me. The first is Alan Kay, his vision for a Personal Computer brusquely put down by CSL Manager Jerry Elkind, falling into a depression. Alan Kay is well known for his brilliance and verbal flourish but Hiltzik does well to also bring home his vulnerability in a way a modern reader would not expect. Kay would ultimately realise his vision of a Personal Computer - with the help of Taylor's CSL - while Elkind was seconded away from PARC on a Xerox taskforce. We tend to recall Kay's assertive (and largely proven) views on Computing. It is unexpected and moving then, particularly with the benefit of hindsight, to see him doubt himself and his ideas before they were fully realised.

The other scene involves Adele Goldberg, co-developer of Smalltalk, and her reactions to Apple's infamous raid on PARC. If you've ever seen The Pirates of Silicon Valley you might have a feel for how this all went down. But Hiltzik's account of it conveys such a sense of dread and hopeless frustration that the movie never came close to recreating.

By the end of the book Taylor's time at PARC draws to a close and with his departure so too does the most storied era of PARC. Scarcely six months after Taylor's forced resignation, the majority of his lab also resign and either follow him to Digital Equipment Corporation (creators of the famous PDP series of minicomputers), or join one of the many startups blooming in Silicon Valley following the success of IBM and Apple's Personal Computer products.

Dealers is fundamentally a story about people that just happen to be in technology - rather than a book about technology itself. It is a human story. It is about what happens when you take the cream of a generation's scientific talent put them in one place and throw lots of money at them. It is about what happens when you combine a visionary maverick with the proneness to credentialism by academically minded administrators. It is about what happens when you have corporate management that want to embrace change but either do not understand it, or worse, fear it.

It is the book's focus on the people of Xerox and PARC particularly, their feelings, motivations and backgrounds that brings this extraordinary tale of modern computing's birth to life.

Google Checkout: Error parsing XML; message from parser is: cvc-datatype-valid.1.2.1: '' is not a valid value for 'decimal'.

I have developed custom modules which add free or bonus items to a customer's cart if they use coupons, meet certain cart criteria or belong to particular customer groups. Buy x, get y rules also work this way. So this is a nuisance. Luckily few customers opt to use GoogleCheckout, but still, I don’t Live with Broken Windows[1].

Chasing the problem down the call stack leads to app/code/core/Mage/GoogleCheckout/Model/Api/Xml/Checkout.php and specifically the _getItemsXml() method.

$unitPrice = $item->getBaseCalculationPrice();
if (Mage::helper('weee')->includeInSubtotal()) {
    $unitPrice += $item->getBaseWeeeTaxAppliedAmount();
}
// ...
<unit-price currency="{$this->getCurrency()}">{$unitPrice}</unit-price>

Now, if the product's baseprice is 0, then for some unfathomable reason it's set to '', not 0. As the unit-price element expects a decimal value, an empty string fails validation.

The fix is pretty trivial

$unitPrice = $item->getBaseCalculationPrice();
if (Mage::helper('weee')->includeInSubtotal()) {
    $unitPrice += $item->getBaseWeeeTaxAppliedAmount();
}

$unitPrice = ((float) $unitPrice > 0) ? $unitPrice : 0.00;

The store I needed to fix only used US dollars so I haven't tested how the use of other currencies or locales might affect this fix.

To apply the fix, don't modify the core codepool, but instead take advantage of the local and community codepool's higher classloader priority[2] and place the amended code in app/code/local/Mage/GoogleCheckout/Model/Api/Xml/Checkout.php.

[1]: 'Don't Live With Broken Windows' is a tip I first read about in The Pragmatic Programmer. It is used to help fight Software Entropy (software's tendency to lose structure over time). This concept has parallels with the real world as urban areas with broken windows tend to see higher levels of vandalism when compared to areas where windows are constantly maintained.

When you ignore small problems it becomes easier to let more significant problems slide too. Hence the rule of thumb, 'Dont Live With Broken Windows'.

[2]: Magento resolves classes in this order local, community then core. This means if two classes have the name Mage_Core_Model_Foo one exists in local the other in core, then the version in local is used.

Thankfully there's a really simple way to socialise Nginx. It is by calling the nginx server command directly with the -s argument instead of using the /etc/init.d/nginx or service nginx commands.

-s lets you send signals to the Nginx master process and Nginx behaves differently whether it receives a quit signal versus a term signal.

# terminate the nginx master process immediately
$ sudo nginx -s stop 
# terminate the nginx master process once all outstanding connections have been completed
$ sudo nginx -s quit 

$ abonner@avalanche:~$ ps aux | grep nginx
root      1063  0.0  0.0  88796  3432 ?        Ss   Jan21   0:00 nginx: master process /usr/sbin/nginx
www-data  9786  1.3  0.0  91564  7352 ?        S    Feb03 190:11 nginx: worker process is shutting down
www-data  9788  1.3  0.0  91288  7072 ?        S    Feb03 189:02 nginx: worker process is shutting down
www-data  9789  1.3  0.0  91160  6956 ?        S    Feb03 190:03 nginx: worker process is shutting down

Let you visitors finish their drink, don't terminate nginx on a production server using /etc/init.d/nginx stop or service nginx stop.

Read more about Nginx's command line options

This year I didn't bother with any. I did in November though, resolve to read more technical books, and to particularly focus on the 'classics'. My motivation was stirred by Panagiotis Louridas's essay 'Rereading the Classics' from the book Beautiful Architecture. In it, Louridas examines the structure of Smalltalk and tries to reason why it achieved greater lasting success blazing a trail for others to follow than as a practical working programming environment.

Louridas suggests that Smalltalk is a classic and by way of justifying why, he quotes Italo Calvino's Why Read the Classics (1986)

The classics [...] exert a peculiar influence, both when they refuse to be eradicated from the mind and when they conceal themselves in the folds of memory, camouflaging themselves as the collective or individual unconscious.

A classic does not necessarily teach us anything we did not know before. In a classic we sometimes discover something we have always known (or thought we knew), but without knowing that this author said it first, or at least is associated with it in a special way. And this, too, is a surprise that gives much pleasure, such as we always gain from the discovery of an origin, a relationship, an affinity.

The classics are books which, upon reading, we find even fresher, more unexpected, and more marvelous than we had thought from hearing about them.

A classic is a book that comes before other classics; but anyone who has read the others first, and then reads this one, instantly recognizes its place in the family tree.

Smalltalk is a language that most of us have heard about but have rarely seen, and after being introduced to it I was inspired to start digging up any further literature I could find. The more that I read, the more that I appreciated its cleverness. Smalltalk can be eerily familiar, and as you begin to grok its syntax it is easy to recognise aspects that have inspired certain features of our 'modern' OO languages.

It is striking that the language itself (and indeed much of the material written about it) is so old. The most modern version of Smalltalk is Smalltalk-80 and although there are modern implementations, the overwhelming bulk of the syntax and environment still cohere to the 1980 standard. Yet download Pharo, read Alan Kay's The Early History of Smalltalk (1993), or Dan Ingalls's Design Principles behind Smalltalk (1981) and it all still seems contemporary. The language itself, which was controversial at the time for eschewing ALGOL syntax, has aged very well. Ruby has maybe made them fashionable, but Smalltalk sported things like block closures over a quarter of a century ago. Reflection, Metaprogramming and dynamic typing - all in Smalltalk in the 70s. Even the idea of using virtual machines to host the programming and operational environment seems remarkably contemporary today, as we increasingly move to abstract our programs and programming environments from bare metal.

It is humbling to see so many ideas that we take for granted today already implemented on a platform that is over a quarter of a century old. It is like amazing at how the Egyptians built the pyramids. Sure we could do it now, no sweat. But we have so much more raw engineering knowledge to throw at the problem. Alan Kay, Dan Ingalls, Adele Goldberg and the rest of the team at the Xerox PARC Learning Research Group designed and implemented Smalltalk with less computing power than my fridge has today. Despite formalising most of the vocabulary for OO software development during the development of Smalltalk it's hard not to feel like some of the energy and innovation of Kay's thinking didn't survive the C++ and Java succession.

Our discipline is still quite young when compared to traditional engineering or the sciences. Yet we seem to keep facing the same problems over and over again. The only difference is perhaps a few orders of abstraction, bigger piles of data and slightly more exotic technologies. But when I think that the fundamental concept of 'Agile' or at least 'iterative' development was doing the rounds in the 60s, it made me wonder what other insights are out there, buried in the forgotten past.

Maybe one reason we tend to forget what others have learned is that the average developer only reads one technical book per year. That means a sizeable percentage of software professionals do not actually read any! I think if I average over my professional working career (after graduating at the end of 2004) I would probably be batting one per year too. If I were to look over the last three years, maybe two per year. I wonder how many mistakes I might have avoided in that period had I read Fred Brooks's Mythical Man Month in 2004 rather than 2012.

My excuse has always been a lack of quality time. Last year in July I left my fulltime job for the world of consulting/freelancing. I imagined it would be easier to read more. In practice it has been, but not as easy as I thought or hoped.

I have a young daughter, my partner has returned to work and despite working from home, I still put in 50-60 hour weeks. It doesn't leave a lot of spare time and what spare time there is, is usually late at night when it's hard to concentrate.

So since I made a determined effort to read more, I've read four books cover to cover, and cherry picked bits out of another four. It feels good and the trick I've found is to read a chapter at a time, whenever you can. Whether it's just before dinner, over coffee/lunch, waiting at the supermarket or just before bed. I found that by reading, even if was just a little bit, every day, I was starting to get through entire books.

It requires conscious effort though, and some material is more suited to this style of reading than others. After a long day just before bed it's pointless trying to delve into SICP. It helps to read certain books during the work day, for example I've been re-reading GOOS and going through Kent Beck's TDD by Example over lunch times. I actively practice TDD so reading a chapter from one of these books midday helps me relate it directly to what I'm working on when I go back into the office.

I have a huge reading list setup in Google Reader, and I'm starting to think it's distracting. I have a slight OCD in that I need to keep the unread count at zero. At the end of a thirty minute work sprint, I would take five minutes to quickly flick through the list. It is distracting actually, and most of the content is superficial. The benefit of reading a book over a blog is the tendancy for a book to have its ideas more fully formed and logically structured. I realise the irony saying this while writing a blog myself. I feel that blogs have their place, but I have been spending far more time reading blogs than reading books. I am now leaving my Google Reader list unread longer and after a few 30 minute sprints picking up a book instead.

As I wrote above, Louridas inspired me to learn more about Smalltalk, and any research into the language leads to Alan Kay's ACM paper The Early History of Smalltalk. The paper beyond providing a wonderful insight into the language and the Xerox PARC is the source of some great quotes. One of my favourites goes:

Where Newton said he saw further by standing on the shoulders of giants, computer scientists all too often stand on each other’s toes.

In Software Engineering, we are so busy looking forward that we don't look back often enough. There is such a rich wealth of knowledge out there already considered and documented. I think we all should make more of an effort to re-discover it.

Here's a small selection I have picked up recently that I didn't know.

sudo !! 

Run the previous command as sudo. This is great when you realise you needed to run something as root.

ctrl-x e

Open up $EDITOR to enter in a long command. In my setup it fires up vim. This is great for some of the long rails commands you need to create controllers or models.

cat /etc/passwd | column -s':' -t 

Column, columnates input, the -t argument will format standard input into a table and -s lets you specify an arbitrary field delimiter. For unformatted input this is very handy.

These next few are specific to zsh, and while I do love bash, since switching to zsh I haven't really looked back. It's things like this that when you work with a terminal every single day, you can't give up.

aaron@tempest ~ $ d                                 
0	~
aaron@tempest ~ $ cd /etc
aaron@tempest /etc $ d
0	/etc
1	~
aaron@tempest /etc $ 1
~
aaron@tempest ~ $  

The 'd' command lists the directory stack, and then entering an integer will switch you directly to the directory index in the stack. It is a killer app.

Moving directories also is very pleasant in zsh. Use '..' to move up a directory, and simply type the name of the directory in, to move into a directory.

aaron@tempest ~ $ ..       
aaron@tempest /Users $ aaron
aaron@tempest ~ $ 

This last one is a trick I've know for a few years, I don't know how much time this has saved me exactly, but I use it every single day.

In vim, if you're editing a file that requires root (or any other user) permissions, you can write the file by doing

:w !sudo tee %

I use it so much that I've set up a leader key binding in my .vimrc

nnoremap <leader>sr :w !sudo tee %<CR>

There's nothing more annoying than making lengthy changes to a config file, go to write it and getting permission denied...

I make all my configs available online at github, if you're interested in seeing how I setup my environment.

To cut a short story, very short, you do this (at least if you have macports in /opt/local (the default), and are using the mysql55 package).

$ gem install mysql2 -- --with-mysql-lib=/opt/local/lib/mysql55/mysql --with-mysql-include=/opt/local/include/mysql55/mysql 

This matters a lot when you want to use xdebug on a project that runs off a virtual machine, or even perhaps a remote server.

The typical pattern (when using PHPStorm in my case) to invoke a remote cli debugging session is to set an environment variable telling the IDE what server configuration to use, and to tell PHP what remote host to connect to.

$ PHP_IDE_CONFIG='serverName=mydevmachine.local' php -dxdebug.remote_host=192.168.0.1 myphpscript.php

Now this will work fine, however if we want to debug during a phpunit test normally you would do this

$ PHP_IDE_CONFIG='serverName=mydevmachine.local' phpunit -dxdebug.remote_host=192.168.0.1 -c phpunit.xml

Unfortunately this doesn't appear to work at the moment (version 3.7.9). If I use the xdebug test client, I can see xdebug trying to connect to the localhost, ignoring what I've told PHPUnit. I'll look into this a bit more later, but I suspect PHPUnit isn't passing on the php.ini settings in a timely fashion for xdebug to hook into.

The solution to this problem is to make use of ssh port forwarding. This works exactly the same for a virtual machine as it would for a remote host, which makes xdebugging on a production machine (hopefully only ever in an emergency!!!) much more straight forward (and less insecure).

$ ssh -R 9000:localhost:9000 myvm.local

This sets up myvm.local to forward all connections to its localhost on port 9000 to the remote client's port 9000. When xdebug goes to connect to localhost:9000, it ends up actually connecting to mydevmachine.local:9000.

It's a bit of a hack, but a time saving one. The other alternative is Vim and its xdebug plugin. This isn't a bad alternative. But once you've experienced the power of PHPStorm's debugging implementation it's hard to go back.

It has been an exciting time to be a PHP Developer these past twelve months, PHP 5.3 is now rock solid and PHP 5.4 is getting there. Both releases significantly modernise elements of the language, closing the gap between PHP and the offerings of more 'in vogue' languages.

In technology we often see change happen in sudden, explosive steps. Often it seems to coincide with developments a technology's ecosystem or among its competitors. For PHP the first major kick was the rapid rise in popularity of Object Oriented Programming in the early 00s. This led to PHP 5's radically overhauled OO implementation in 2004. The next kick, I feel, came in 2005 when Ruby on Rails exploded into everyone's consciousness. RoR provided a full stack web development platform that drastically simplified creating complex web applications. The PHP community responded in kind with a number of 'fullfat' Model View Controller (MVC) Frameworks, the most successful being Zend's and Symfony.

It all depends

The arrival of PHP 5.3 and features like Namespaces, PHAR, Closures and the ubiquity of Github is having the effect of giving PHP a new kick, and the results are starting to make themselves felt. We now have second generation frameworks from Zend and Symfony leveraging these technologies.

One problem remains though, and that is managing and distributing dependencies. Modern web development platforms all now have robust dependency management tools available and in the PHP camp, PEAR wasn't really cutting it.

The success of Symfony2 in particular, with its emphasis on high quality, modular components, forced PHP developers to address how they bundled and distributed library code.

Luckily for us, the guys behind Composer, (again, taking considerable cues from the Ruby community) have licked it. Composer, in tandem with Symfony2 components allow PHP Developers to confidently build on top of other developers' libraries.

Do we really need another packaging tool?

Why did we need another package and dependency management tool anyway? What really, is wrong with PEAR? Well, if we wind the clock way back to 1999 when Netscape Communicator was still the most popular web browser and Google had just moved out of Susan Wojcicki's garage, PEAR was conceived as PHP's answer to PERL's CPAN. Despite some strident efforts, it never really managed to become the most pleasant package manager to work with: rigid, elitist and worst of all, difficult for end-users. PEAR's age strictly speaking is not the problem, but its centralised nature is a bottleneck and there is no straightforward way to handle two packages with varying dependencies. For example: say package x, requires stable package y. Package z requires beta package y. You can't install both. Dependency and package management has moved on a long way since 1999.

Over time PEAR's shortcomings have led to a graveyard of abandoned packages, code of at best variable and at worst, dubious, quality, and a community lacking in any sort of dynamism. If you make something easy, people will use it. PEAR is difficult to use for developers and users alike.

Composer democratises (in the best sense) things and puts full control of dependencies in the hands of library developers. Free to pick and chose code they want to use. Free from having to worry about navigating the PEAR jungle. Here the rise and rise of Github has been key. Composer can sit over the top of code distribution services provided by Github, or it can use its default Packagist repository. This removes the need for libraries to live in a blessed canonical repository or for developers to host it themselves.

... profit?

There's no compelling need now to constantly rewrite basic library components (I think we've finally licked what ought to be the basic issue of class loading!), Free of the shackles of PEAR, we are witnessing an explosion of high quality PHP frameworks, libraries and utilities.

PHPSpec, Behat, Twig, Mockery, Doctrine, are just a few that immediately spring to mind. Some (such as Doctrine) have been around a while. However the advances PHP 5.3 brought to the table have significantly helped improve the utility of these projects.

Anyway, so (after a fashion) I come to the tool that motivated me to write this post, n98-magerun.

The name is horrible, but the tool itself is brilliant. In short, it's Drush for Magento and it's wonderful. It is one of those tools that makes you wonder what on earth you did before it.

I have a folder full of bash scripts, cobbled together to help automate the mind-numbing process of managing Magento installations. Over the course of a few months Christian Münch and friends have overseen a small tool quickly develop into the kind of utility we've all wanted but never had the time/patience to build ourselves.

Magerun is elegantly simple for the user, and cleanly extendable by developers. It is a perfect illustration of why it's such a great time to be a PHP developer. Better dependency management, easy distribution, modular libraries and powerful language syntax have all came together to let someone with an itch, scratch it quickly and effectively.

It has become several orders of magnitude easier to develop, package and distribute PHP libraries and utilities. The result of this leap forward is a brilliant tool that helps Magento developers dramatically increase their productivity.

I'll be writing about how awesome Magerun is shortly, but just one of its cool features is the ability to dump out a merged version of Magento's config.

This is extremely helpful when trying to resolve conflicts between modules, or figure out what bit of configuration is taking precedence.

The resulting xml though is pretty raw and unformatted, but xmllint can fix that.

Xmllint expects a file to work with and cannot use piped input. So we need to use bash's Process Substitution feature to avoid having to create temporary files.

$ xmllint --format <(magerun config:dump)

So, magerun and xmllint, a simple way to get a formatted, easy to examine view of how Magento is putting your install's configuration together.

$ git show --name-only <commit>

This will give you a list of affected files in commit

Alternatively if you don't care what differs in the specific contents between two branches, and only want to see different files you can do

$ git diff master..origin/master --name-only

This will show you the list of files that are different between your local master branch and the remote master branch. Handy if you have just done a git fetch and want to see what's different before merging or rebasing.

First thing, shut down the running instance and then restart it directly

$ sudo -u <mysql_user> mysqld_safe --skip-grant-tables --skip-networking

The --skip-networking option is important, as by skipping the grant tables, any user can connect to the running mysqld service, will full permissions.

Once you've started the server up, login without a password, and issue an update query to the mysql.user table.

$ mysql -uroot mysql
mysql> UPDATE user SET password=password('newpassword') WHERE User = 'root'

Close down mysqld and restart. You're good to go.

We've seen some pretty epic PHP rants this year, probably the most famous among them are PHP a Fractal of Bad Design, and Jeff Atwood's latest (in what seems to be a biennial broadside) The PHP Singularity.

The common thread in these rants is incredulity that anyone would, in 2012, write new code in PHP. There's a lot of reasons why someone might write greenfield PHP code in 2012. But equally (and this is said as a decade long PHP programmer) I have to admit that plenty of the criticisms levelled at PHP are valid. Yet, for the most part they just don't, particularly, matter.

One criticism that is wholly invalid yet comes up time and time again, is that using PHP intrinsically leads to bad code. I don't feel this is an inherent trait of PHP itself, so much symptomatic of the popularity and low barrier to entry of PHP. Basically there's more examples of bad code out there compared to pretty much any other platform because there's simply more code out there, written by programmers of wildly varying skill. The other problem, is that PHP came into existence as a scratch to a C programmer's itch. PHP was developed at a time where people still actually wrote web applications in C and where the stateless nature of HTTP was relatively respected. PHP, like the web itself, has moved on dramatically since then.

A modern PHP 5.4 webapp looks about as similar to an early 00s PHP 4 webapp as Scala does to Java. Yet many critics when slating the language appear to be code archeologists, excavating pre-historic practices that went out of favour long-ago.

This can be partially forgiven, because owing to the age of the language there's plenty of out of date information out there with high rankings in Google. The ubiquitous w3schools is an unfortunate example of bad practices coming well ahead of sites with more modern approaches to solving problems in PHP.

So 'New PHP' is very different to 'Old PHP'. But the 'Old PHP' is what most people seem to find when searching in Google and this confuses people.

We see this manifested in blogs like Will Rails become the new PHP. This blog has a number of spectacular shortcomings, most egregiously the author's horribly naive view of the PHP community, but the interesting one is his ignorance of the power of Google.

There's plenty of support out there for budding PHP programmers, whether on the web, on forums, or IRC. There are countless, well attended, supported and growing conferences, meetups and the like for PHP programmers. One thing that a community cannot do, is force Google to nuke w3schools' PageRank. Which means that brilliant efforts like PHP The Right Way get swamped by old, incorrect and at times dangerous dreck.

And what the 'Will Rails become the new PHP' author perhaps hasn't realised, is Rails, at least in the terms he's trying to couch it, has already become the 'New PHP'. I am a novice Rails developer, I like to hack around in it as it can be quite fun to spike out solutions. What I've been struck by, is the sheer amount of bad advice out there. Advice novices will come across, if they turn to Google for help.

If you're well versed in a platform you learn through brutal experience what works, what doesn't and your nose is finely tuned to bullshit. When I read a PHP article I know instinctively if what I am reading is reliable. But with Rails, as a novice, I don't quite have that sense, beyond my own background experience with other programming languages.

So let's look at an example. I've been working on a dead simple Rails authentication webservice. It listens for HTTP requests for /login, /logout, /session, etc., and emits either XML or JSON in response. I'm using the respond_to method to serve out these responses. Unfortunately what I found is if I request a route that does not exist, I get an HTML error back. This doesn't make a lot of sense for a webservice that otherwise speaks XML and JSON.

Other global exceptions similarly respond with HTML. I don't want to wrap every action up in a begin/rescue block and there is certainly no way to intercept router exceptions in actions anyway. So I needed to learn how to catch global exceptions.

In my journey of (Google) discovery I came across this blog post and appeared to hit paydirt. The advice appears to be legit, hell someone in the rails community even featured it in a podcast. So on the face of it this seems good. But that switch statement sure is smelly. Does it, really, need to be this hard?

Now my general purpose programming brain recognised that this code while solving my problem, is not ideal. And why is it not ideal? This one method sure has a lot of responsibility. Method names with plurals in them are usually a code smell. Over time as more specific exceptions need to be handled I would end up with code that is as easy to read as Goethe's Faust, photocopied and in the original Gothic script (not easy). What we have here is a God Method in training.

Now, what if I am a Ruby/Rails/Programming novice? I have got plenty of other stuff to learn, I'm going to go right here and Cargo Cult this code into my webapp and move on. Just like all the rookie PHP coders do, right?

Well, I didn't do that. I saw that this rescue_from method was pretty awesome and so I went to the Rails API docs to look it up.

API docs for any language are pretty terse, but what jumped out at me was this line:

“Handlers are inherited. They are searched from right to left, from bottom to top, and up the hierarchy. The handler of the first class for which exception.is_a?(klass) holds true is the one invoked, if any.”

This isn't great documentation admittedly, but it means basically, if you put rescue_from Exception at the bottom of the list of rescue_from handlers in your application_controller.rb file, then since everything derives from Exception, nothing else will get a look in (Rails will look at the handlers from the bottom up).The author of that helpful blog we found didn't realise this, and so his solution was needlessly complicated.

What can we learn from this? Well Rails programmers certainly live in a glass house and shouldn't throw stones is one thing. But on a slightly less trollish note, there is a problem here for all novice programmers that turn to Google to help them solve problems. The answers on Google are usually either wrong, or at best, incomplete. As the web gets older, bad and out of date advice piles up making it much harder for novices to find good advice.

Knocking a language for this phenomenon (or a framework, seriously, whatever) is more than a little ignorant and doesn't solve the problem. Efforts like PHP The Right Way is how PHP is trying to fix it. If Rails really doesn't want to be the 'Old PHP', they need to realise it's less to do with languages and platforms, and more about SEO.

To see what branches your local repo things exists you do something like this:

$ git branch -rv
> origin/1620-upgrade  2e0cc56 Ignore active local.xml from vc
> origin/HEAD          -> origin/master
> origin/cas-sso       2351be5 Add gateway logiin and logout support
> origin/giveaways   63daf5a Use cms blocks for banner placements
> origin/master        496c975 Merge affiliate module
> origin/newskin      d7220c9 Optimise skin and ui images
> origin/release       496c975 Merge affiliate module

So this is my local Magento git repository, many of the branches here are now defunct and no longer in the remote (i.e. I had previously had used $ git push origin :branch from another host)

To refresh then I need to prune my branches list. The git incantation to do this is

$ git remote prune origin
> Pruning origin
> URL: dev@vcs:git/store.git
* [pruned] origin/1620-upgrade
* [pruned] origin/giveaways
* [pruned] origin/newskin

Looking at the remote branch list again:

$ git branch -rv
> origin/HEAD    -> origin/master
> origin/cas-sso 2351be5 Add gateway logiin and logout support
> origin/master  496c975 Merge affiliate module
> origin/release 496c975 Merge affiliate module

A simple configuration recipe for adding new cache tags to the Magento backend’s “clear cache” feature.

In theory the Magento cron subsystem should keep a lid on these tables growing too big, but I avoid using Magento cron, preferring to handle that myself directly via crontab tasks.

The other option is to write your own table cleaning script (or copy one from somewhere), and this will work too. But it's annoying, if you don't want this log data, why write it in the first place.

So my solution is to disable it by removing the observer events that perform the logging.

I have this in my local.xml which takes precedence over other nodes in the config and therefore overwrites them. Here, by setting the observer to be the string 'disabled', the existing observer event is removed and replaced with something that will never be fired.

Now, you don't need to worry about periodically cleaning out your database, nor do you need to fear a 3am text message from your production DB servers screaming about the disk being full...

I'm going through some PCI scan results this morning, and in the main it's going well, but I got a couple XSS hits on our catalogsearch pages. This is odd, I think. I've audited these pages, they definitely get routed through magento's escaping code.

On closer examination it turned out the form was okay, it was via the breadcrumbs, that unescaped input was getting into the wild.

I'm running Mage 1.6.x so this code may look a little different if you're running 1.7

Take a look at app/code/core/Mage/CatalogSearch/Block/Result.php, and specifically at the prepareLayout() method:

Now if you look at line 11, if breadcrumbs are enabled, unescaped input is happily added ready for output.

$title = $this->__("Search results for: '%s'", $this->helper('catalogsearch')->getQueryText());

This fix is easy, replace line 10 with:

$title = $this->__("Search results for: '%s'", $this->helper('catalogsearch')->getEscapedQueryText());

This is a really neat example of the evils of duplication and where bad programming practice can lead to real world problems. I am speculating, but it seems reasonable to infer that the original programmer got trigger happy with the copy & paste keys. Later, at some point you could imagine another engineer coming in to XSS safe the code fixed one bit, but (and programmers are human) missed the other (exactly the same line), and we end up with an issue like this.

Personally, I patched the file as described above and stuck it in app/code/local/Mage to override the core code pool version.

Unfortunately this doesn't happen very often and when using a stack of Rubygems, you almost always get bitten by something.

My cause for complaint today is Vagrant and Chef, well specifically Chef Solo. Vagrant is fine, it does what you tell it to do, but for most use-cases Chef Solo is the right tool to use for provisioning your virtual server. The Vagrant docs on Chef Solo unfortunately fib you, they say you can use Data Bags with Chef Solo, but by default you cannot.

This is a big deal as many useful Chef recipes make heavy use of Data Bags. Data Bags which let you provide environment specific configuration for your provisioning is not yet supported by the stock Chef Gem (currently version 10.12.0). In order to make use of Data Bags with Chef Solo, you need version 10.14.0 and above. This means building the gem from source.

I use Veewee to build my vagrant base boxes (you should too, it's awesome!), and you can edit the postinstall.sh file in your box definition folder to build Chef from source, rather than installing it directly via Rubygems.

You can repeat this for your local dev machine, and now you can get Chef Solo cooking up your recipes and happily using data bags.

This overrides the core code pool's adminhtml definition, and puts a dependency on a non-existent module. Effectively, this disabled the menu item because it no longer meets the defined dependency requirements.

As always with any magento configuration / module changes, you may need to clear caches for this to take effect.

Yesterday I had to do an emergency restart of mysql in production and the resulting magento report/xxxx files swamped out everything else that I might have wanted to look at.

So specifically, I wanted to delete all the files that were created between a start and end date.

GNU find makes this easy

$ find . -type f -newer 655958024588 ! -newer 627812220533 -ls

This instructs find to list (-ls) all files (-type f) that are newer than a file called 655958024588 (-newer) and not newer than 627812220533 (! -newer).

If you do not have two files to act as date range boundaries, you can use touch to create them.

$ touch -t yyyymmddHHMM start_date_file
$ touch -t yyyymmddHHMM end_date_file

Then supply these file names to -newer and ! -newer.

To delete the files we can use -exec.

$ find . -type f -newer 655958024588 ! -newer 627812220533 -exec rm -rf {} \;

Here it's the -exec argument does the heavy lifting. {} is a placeholder for the file name (find substitutes '{}' with each found filename) and ; terminates the command sequence (much like it does in regular bash).

So, if you want to use veewee to setup a new amd64 Precise Pangolin basebox for vagrant, you either have to pull the latest veewee sources from github, or download the most recent releases templates and copy them over into your veewee gem folder.

This is where veewee-templates-update steps in, it automates that latter step (downloading and installing just the updated templates) for you.

Installation is simple:

$ gem install veewee-templates-update

Then, just run the updater:

$ veewee-templates-update
> Veewee: /home/aaron/.rvm/gems/ruby-1.9.3-p194/gems/veewee-0.2.3
> Downloading: https://github.com/jedi4ever/veewee/tarball/master
> Extracting: CentOS-4.8-i386 CentOS-5.5-i386-netboot CentOS-5.5-x86_64-netboot 
> ...

$ vagrant basebox define precise-amd64 ubuntu-12.04-server-amd64
> ...
> (profit)

Anyway, quite happy with it so far, but like any geek with new kit, I want to know everything about it, and make it dance to my whim.

Two things learned this morning: if you have an old Magsafe (1) powerpack, which I did from my old 13" Mac Pro, and it has higher or equally rated wattage you can use it with your macbook air.

It makes sense, a high rated powersupply can support a lower power rated device, but not vice versa. So a 60 watt Macbook Pro magsafe can power a 45 watt Macbook Air. But a 60 Watt Magsafe can't power a 15" 85 watt macbook pro.

The other thing which I have discovered, is that the thunderbolt port on the newer Airs use the same socket form factor as the mini display port. This means is if you have an old Mini DisplayPort -> DVI adapter lying around, you can reuse it.

My first real job has been to get a decent unix environment up. OSX's BSD utilities don't really cut it. Macports is far and away the best distribution out there.

Once you install coreutils and get ls, find etc it makes sense you will want to change your shell to a modern version of bash (or zsh if that's the way you roll).

Lion ships with bash 3.2 whereas Macports will give you a contemporary version 4.2 Unfortunately it's not as simple as going

$ sudo port install bash
$ chsh 
  <input /opt/local/bin/bash>

I had to do this before, but I'd forgotten there's a trick to changing your shell in OSX to a non-standard location. The file /etc/shells contains a list of valid shells chsh will permit. You need to edit (as root or via sudo) this file and add your macports shells. Once that's done chsh will let you change no problem.

To take a column from a MySQL database and quickly output it ready formatted as a Javascript array literal (without any specific escaping) do:

echo 'SELECT column FROM table WHERE some_column = "somevalue"' | mysql -uuser -ppass --silent yourdb | awk -v q="'" '{ print q $0 q }' | paste -s -d ',' | sed 's/(.*)/[\1];/'

The first part of the command is self explanatory, you pipe in a query to mysql, and ask it to give you raw unadorned output. It will return each row for column 'column' from table 'table' as a line of output.

You pipe it to awk and ask it to wrap the values in single quotes. Due to shell escaping with single quotes, you set the q variable to a single quote. Paste then joins all the output lines together separated by commas.

Finally I use sed to wrap the resulting output in Javascript array literal '[' and ']' symbols. Awk or any other tool to concatenation approach would do just fine here too.

field_name: [Start TO Finish] 

You can also use wildcards and specific constants in a logical way e.g:

[NOW TO *] or [* TO *]

To search over documents that do not have a value for that date field, e.g. is NULL, you use the syntax:

-field_name: [* TO *].

It is hard though, to search for dates that are EITHER NULL OR lie in a specific range.

It would seem logical to specify

date_field:[Start To Finish] OR -date_field: [* TO *] 

Unfortunately Solr does not appear to support specifying a field multiple times in this way.

So the trick is to effectively query for everything you do not want, then negate the result.

This approach says select me anything that isn't in the range of this date and is not null. When you invert that result, you get all the documents sit inside the date range or are NULL.

The query to weave this magic is

-(-date_field:[Start TO Finish] AND date_field:[* TO *])

A Singleton, if you've not heard the term before, was popularised in the Design Patterns book by the Gang of Four (Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides). To be very succinct, a Singleton is a way to ensure there is only ever one instance of a class in an Object Oriented design. To put it in even simpler terms , it is an Object Oriented version of a global variable.

It's used heavily in Magento (in the app/code/core directory, 2261 times in fact!). But anyway, why is it considered harmful? There are a number of arguments for why and why not. Herb Sutter's Once is not enough gives a pretty good (and fun to read) overview of them, or you can read Kenton Varda who looks in-depth at the topic. I generally think though, that in Object Oriented software, you're seeking to create abstractions around complexity. The Singleton is a (too) convenient escape hatch from encapsulation and can lead to the attendant issues you get with global variables.

In the Magento/PHP land a more implementation specific problem with Singletons, is memory consumption. Today I was revisiting a Magento Promotions extension I had written and trying to figure out why it was suddenly obliterating PHP's memory_limit.

This extension basically piggybacks on the existing Promotions/Coupons system, but generates an index of products that match coupon codes, the price before and after the promotion is applied and some other metadata.

In order to determine what products have a promotion associated with them, I run through all the products, and match SalesRule conditions against them. I create a synthetic quote for the products that match and then pump them through the SalesRule validator. This effectively applies the promotion to the product and let us see what the savings are.

It's fairly basic, it doesn't look at multi product combinations but it works well enough for simple cases.

/**
 * Apply pricing rules to a synthetic quote to calculate discounted price
 * 
 * @param string $couponCode
 * @param Mage_Catalog_Model_Product $product
 * @return  float
 */
public function applyToProduct($couponCode, $product)
{
    $quote = Mage::getModel('sales/quote');
    $item = Mage::getModel('sales/quote_item')
        ->setQuote($quote)
        ->setProduct($product)
        ->setQty(1)
        ->setBaseDiscountCalculationPrice($product->getPrice())
        ->setDiscountCalculationPrice($product->getPrice());

    $validator = Mage::getSingleton('salesrule/validator')
        ->init(1, 1, $couponCode);
    
    $validator->process($item);
    
    return $product->getPrice() - $item->getDiscountAmount();
}

Now when I wrote this code, it seemed sensible to use the validator as a Singleton, after all I only needed one copy of it. It didn't, at the time, seem to make sense to create and then destroy the validator a couple of thousand times during indexing. Indeed when this code was first deployed, everything ran smoothly.

Recently the user of this extension added a whole bunch of sales rules - and this caused that product/salesrule index loop to detonate.

That Singleton Validator, which was written as some sort of optimization, started happily hosing over a gig of ram.

Changing getSingleton() to getModel() took ram usage down from 1100MB to about 80MB.

My suspicion is that PHPs garbage collection wasn't cleaning up adequately after each validation attempt. As the validator is effectively static, it never gives up its references for PHP to clean up. When you use getModel(), the validator loses all its references after each loop. While it means it also has to be constructed after each loop, but that allows PHP to free the memory it is using.

The Singleton is already a controversial pattern these days, but Magento developers should be particularly wary of it's implementation and its scope to hose memory.

Typically I manage PHPStorm by extracting it to /opt and then symlinking the extracted folder to /opt/PhpStorm.

To create a nice launcher for Unity you create a desktop entry under ~/.local/share/applications

$ vim ~/.local/share/applications/jetbrains-phpstorm.desktop

Now paste the following in (adjusted for your own paths)

[Desktop Entry]                                                                  
Version=4.0.1                                                                   
Type=Application                                                                
Name=JetBrains PhpStorm                                                         
Exec=/opt/PhpStorm/bin/phpstorm.sh %f                                           
Icon=/opt/PhpStorm/bin/webide.png                                               
Comment=Develop with pleasure!                                                  
Categories=Development;IDE;                                                     
Terminal=false                                                                  
StartupNotify=true                                                              
StartupWMClass=jetbrains-phpstorm           

Hit your windows or super key and type in 'Php' - you should see the newly created desktop entry there. Once launched and fully started, you can opt to keep PHPStorm locked to the Unity launcher for easy startup.

Even today, almost all x86 linux distributions can run (in theory) on a 32bit 386 processor. Let's be clear though, while it is a remarkable (and successful) processor, the 386 is an antique. The selling point of Gentoo was that it was one of the few distributions to give you the power to build a system, specifically for modern processors abandoning backwards compatibility for long-obsolete ones.

In the early days you had to start from what was called a (and I don't think they do it this way anymore) 'stage 1' install. A stage 1 install is where all you have is a bootable livecd with a few basic set of tools: a c compiler, a shell and the basic gnu coreutils...and irssi. Enough utilities to allow you to build further C packages. You'd run the command 'emerge system' and it would go off and build gcc, glibc, coreutils etc. Once these were built you'd then rebuild glibc, and gcc with your newly compiled, architecture specific, compiler and c library.

Anyway, this took a long time, particularly on sub gigahertz Pentium 2s and 3s, and Gentoo systems tended to break a lot, and by break, in the absolute best case, I mean merely became un-bootable.

The process to recover the system was pretty much the same as to install it, you had to boot from a livecd, configure your network card, hook up to the network, then chroot to the broken disk. At this point you could try and repair whatever damage you had caused.

These days if you do something silly, like I don't know, try and dist-upgrade from Ubuntu Oneiric to Precise, you can get that true Gentoo feeling (i.e. nothing works and you can't boot the machine).

This happened to me this afternoon and the hard yards done with Gentoo came to the rescue.

Here's how you do it.

Boot up from a livecd (or usbkey), get the network card modules loaded and get a dhcp address. With the Precise Live DVD you can do all of this pretty easily by selecting the 'try without installing' option from the bootloader.

Once you're online, you need to prepare the mount. First step is to mount the root partition somewhere, I typically use /mnt/ so say /mnt/oneiric (it can be whatever). If you're not sure of your partition numbers, your livecd will almost certainly come with fdisk with which you can push 'p' to print the partition table.

$ mount -t ext4 /dev/sda5 /mnt/ubuntu

If you have a separate boot partition mount that too.

$ mount -t ext2 /dev/sda1 /mnt/ubuntu/boot

Now in order to have a functional chroot, we need the proc, dev and sys subsystems to be mounted onto the chroot. This is the tricky bit.

$ mount -t proc none /mnt/ubuntu/proc
$ mount -o bind /dev /mnt/ubuntu/dev
$ mount -o bind /sys /mnt/ubuntu/sys

In the case of the sys and dev dirs, we need to reference the exact same mountpoints as the host so we use the -o bind option.

Last thing, we want to have functional network name resolution so we copy over the host's /etc/resolv.conf to /mnt/ubuntu/etc/resolv.conf

Now the chroot is ready

$ chroot /mnt/ubuntu /bin/bash
$ source /etc/profile

The chroot will be pretty much as it would be if you'd booted into it normally with a few exceptions. The kernel and kernel modules will be those of the host. If you need to access some specific hardware you need to set this up on the host.

My busted Precise install was fixed with a simple apt-get update and upgrade, before re-running the grub installer.

Forcing the cataloginventory_stock indexer to re-run fixes the situation but if you want to script the status update of many stock items, you can have a short period where your store's products will be unavailable.

Stepping through the issue I found myself in app/code/core/Mage/Catalog/Model/Resource/Product/Status.php and specifically the getProductStatusMethod()

/**
 * Retrieve Product(s) status for store
 * Return array where key is a product_id, value - status
 *
 * @param array|int $productIds
 * @param int $storeId
 * @return array
 */
public function getProductStatus($productIds, $storeId = null)
{
   $statuses = array();

   $attribute      = $this->_getProductAttribute('status');
   $attributeTable = $attribute->getBackend()->getTable();
   $adapter        = $this->_getReadAdapter();

   if (!is_array($productIds)) {
       $productIds = array($productIds);
   }

   if ($storeId === null || $storeId == Mage_Catalog_Model_Abstract::DEFAULT_STORE_ID) {
       $select = $adapter->select()
           ->from($attributeTable, array('entity_id', 'value'))
           ->where('entity_id IN (?)', $productIds)
           ->where('attribute_id = ?', $attribute->getAttributeId())
           ->where('store_id = ?', Mage_Catalog_Model_Abstract::DEFAULT_STORE_ID);

       $rows = $adapter->fetchPairs($select);
   } else {
       $valueCheckSql = $adapter->getCheckSql('t2.value_id > 0', 't2.value', 't1.value');

       $select = $adapter->select()
           ->from(
               array('t1' => $attributeTable),
               array('value' => $valueCheckSql))
           ->joinLeft(
               array('t2' => $attributeTable),
               't1.entity_id = t2.entity_id AND t1.attribute_id = t2.attribute_id AND t2.store_id = ' . (int)$storeId,
               array('t1.entity_id')
           )
           ->where('t1.store_id = ?', Mage_Core_Model_App::ADMIN_STORE_ID)
           ->where('t1.attribute_id = ?', $attribute->getAttributeId())
           ->where('t1.entity_id IN(?)', $productIds);
       $rows = $adapter->fetchPairs($select);
   }

   foreach ($productIds as $productId) {
       if (isset($rows[$productId])) {
           $statuses[$productId] = $rows[$productId];
       } else {
           $statuses[$productId] = -1;
       }
   }

   return $statuses;
}

This method goes through a list of productIds, and will assign a status id to them, this is typically used on grouped products when determining if all their children stock items are out of stock.

In testing, the status ids were all coming back as -1, i.e. not valid, and so therefore the group was out of stock.

In my code the store id was neither null, nor the default store id, so execution fell through to the else branch. At first I inserted a print_r($select->assemble()) to see the SQL being generated. The SQL was fine and when pasting it into MySQL I got a bunch of valid looking results. Funnily though, the status column was first, and the product id column was second (unlike the if branch, where they were in reverse order). This presents a problem when we reach the fetchPairs() statement.

Zend DB's fetchPairs returns an associative array resultset where column a is the key, and column b is the value. Because the SQL was returning the status column first (i.e. the key value) the result set consisted of just 2 rows (for each unique status code). In order for this code to work as you would expect the entity id (product id) needs to be first in the result set, then it gets used as a key.

The fix is straight forward enough, replace

$select = $adapter->select()
            ->from(
                array('t1' => $attributeTable),
                array('value' => $valueCheckSql))

with

$select = $adapter->select()
            ->from(
                array('t1' => $attributeTable),
                array('entity_id', 'value' => $valueCheckSql))

This way the product id is always used as the key in fetchPairs and you get a status result for each product.

• http://www.growing-object-oriented-software.com/
• http://www.mockobjects.com

The big idea is messaging. Kay98. It's a quote cited early on in Growing Object Oriented Software Guided by Tests (GOOS), a book that looks at Test Driven Development (TDD) using Mock Objects. This idea of messaging being central to Object Oriented Analysis and Design (OOAD), drives much of what is presented throughout the book.

I think as OOAD has matured over the past decade, the mode of thinking of classes as hierarchical constructs has lost favour. Increasingly OOAD is about managing the collaboration of a large number of small, independent, objects. In such designs the solution to the problem is achieved by the way the developer defines the software's object graph – i.e. its composition. In a design that is focussed on getting the composition of objects just so, communication between them is the important thing, much more so than classification.

In GOOS, TDD is presented as an exercise to first understand and then improve messaging protocols between objects. GOOS demonstrates that Mock Objects are an ideal tool to help discover these protocols . This is unusual, even in 2012, for a lot of developers. Typically I've always used Mock Objects as a Test Stub or Double. A placeholder object to induce some specific behaviour I want to test, or to isolate the unit of code I'm testing. GOOS sees Mocks a different way, as a means of representing roles within a system. The authors say a Mock Object is not a Stub, but instead an interface to some behaviour or role. Mock Roles not Objects, a paper written by Freeman, Pryce, et al. way back in 2004, explains this concept and sets out much of groundwork for GOOS.

GOOS itself is structured into five sections. The first couple of sections (very) briefly introduce the reader to TDD, the basic tenets of Agile development (very heavily influenced by Extreme Programming, XP), testing tools (JUnit, JMock, Hamcrest, Windowlicker) and the authors' OOAD philosophy.

The overwhelming impression I get from Freeman and Pryce's introduction to TDD and OOAD is that they see writing tests as less an exercise in producing a regression catching suite, and more as a design exercise. By writing a failing test up front, you have an immediate, testable, statement of intent as to what the software will do, and just as importantly, what it wont do. Focussing on just a narrow slice of behaviour, as represented by a single test, helps narrow the scope of what needs to be done. The act of satisfying the test, making it pass, focuses the developer's mind on the domain of the problem and forces both the developer and the customer to think of what from the environment impacts the test. Flushing out dependencies, object peers and services early on is a good thing.

Too often in Big Design up Front approaches, you write a ton of code according to a pre-determined (but unchecked) idea of the environment. When you go to hook everything up at the end you can find that, to the horror of your project sponsor (and spouse who wont see you for at least the next week), that, actually nothing hooks up. Or worse, you have misinterpreted what the Sponsor wanted. In a Test Driven Design, if elements of the system are incompatible, you're alerted to it early. If you have gotten the requirements completely wrong the customer can see it straight away. TDD takes 'Fail Fast' to its logical extreme.

The meat of the book is a worked example. The authors describe a fictional auction sniper system that connects to an auction, makes bids on items and either wins or loses. It's a simple example, but complicated enough to run into common issues developers face when developing OO software. Certainly there is little trouble filling out 150 pages as the authors work from an abstract set of stories to concrete code. What is good about their example is the way you see the code transforming in stages as extra features are added. The writers are careful to explain the motivations for the transformations they make and they tie it back to the TDD and OOAD principles they introduce in the first two sections.

Having worked through the Auction Sniper application using Java, JUnit, JMock, Hamcrest and Windowlicker there's a brief recap and the book moves swiftly onto it's fourth and fifth sections.

The fourth section covers 'sustainable TDD', an important and increasingly relevant topic for many developers. The burden of maintaining poorly designed test suites is a drag on developer productivity. Rather than liberating developers to improve the structure of their code, bloated indecipherable test suites become a handbrake. GOOS goes through techniques to keep test suites effective, flexible and importantly - expressive. The concept, that software is about communication, is emphasised across the book. Tests are no different. Tests should express the developers intent and function as the rough and ready documentation of a unit of code. I found the advice around constructing data builders - techniques for creating test data for use in your test cases - particularly valuable.

The final section tackles the really hard stuff, dealing with persistence (and by extension any sort of frameworky data service), asynchrony and lastly concurrency.

I found GOOS easy to read and its chapters are of a length that can be easily read on the train/bus or in short bursts. Physically, the paper is of high quality and the typesetting clear, and easy to read. I really like the images and diagrams which are simple, authentic and aid what is being discussed. I felt like the introductory sections to TDD and particularly the author's OO philosophies were fairly succinct, perhaps too succinct. But as they state from the outset, this is not a book on TDD. And anyway, Meszaros07 should be enough TDD for any human being.

GOOS comes with a particularly outstanding bibliography. Freeman and Pryce's academic backgrounds and broad reading are on display with the depth and quality of their references. As a starting point GOOS, while being a pretty domain specific (Mock Object) text, serves as a wonderful launching pad to further OOAD and TDD reading.

I like that the authors are people who practice what they preach. GOOS is a book for people who write real code in the real world. Sadly there seems too many authors, 'consultants' and coaches these days that talk a lot about programming and programming techniques yet seldom practice it in the wild. GOOS reads like a book borne out of brutal trench warfare with Objects. It is refreshing to read a book detailing a principled design philosophy and practice that has been tested in the dirty unwashed world of Enterprise

I've slowly been pruning back my physical book collection and trying to maintain a library of what I would consider 'classics'. The GOF book, Fowler's Refactoring and PoAEE Books, Beck's XP White Book, the Prag Prog Book, Kernighan and Richie's C Book, SICP and Knuth's The Art of Computer Programming series (and if I can ever get it, Mike Abrash's Black Book). I think, for me, this book fits into that category. It represents a decade of work and thinking, neatly explained by two highly skilled and above all practical developers who were at the heart of it all.

I highly recommend this book to anyone actively practising TDD and also generally to anyone with an interest in Object Oriented Software Design and Practice. While GOOS is a 'Java book' and day to day I program in PHP, the principles, practices and overarching philosophy easily translate.

Follow Steve Freeman @sf105 on Twitter, and read his blog http://www.higherorderlogic.com/. Nat Pryce is @natpryce and blogs at http://www.natpryce.com/.

Also be sure to check out the excellent mailing list http://groups.google.com/group/growing-object-oriented-software.

To map it to twig, edit vimrc and add:

au BufRead,BufNewFile *.twig set filetype=htmljinja

Generally speaking (i.e. not with Mac OSX's Terminal.app) .bash_profile gets sourced only on login. Specifically this means only when you enter your username and password from the console. The .bashrc file is sourced when starting an interactive session, that is, whenever you open up a terminal.

There is some confusion here, when you open up a login shell, such as if you use the su - command or run an explicit login shell sometimes provided by a desktop environment. In these cases the rule applies, a login shell means .bash_profile is sourced first, then .bashrc.

I tend to put environment setup in bash_profile, things like paths and any specific one-time configuration settings that aren't likely to change very much. But it's quite reasonable to just put a source .bashrc in your .bash_profile and then put everything in .bashrc.

Then, to include it in a template (say homepage.phtml):

<?php echo $this->getLayout()->createBlock('cms/block')->setBlockId('indentifer')->toHtml() ?>

  $mock = $this->getMock('Zend_Mail');
  $mock->expects($this->any())
            ->method(new PHPUnit_Framework_Constraint_IsAnything())
            ->will($this->returnSelf());

This has the effect of making any method call on your mock object return a reference to itself.

Are software patents really protecting innovation? Or are they really just protecting large businesses, increasing barriers to entry and reducing the ability for a small, innovative competitor to survive?

It's something [software patents] that's really depressing because it's so horribly wrong for someone who's a creative engineering type to think about patenting a way of thinking. Even if you had something really, really clever, the idea that you're not going to allow someone else to follow that line of thought... All of science and technology is built standing on the shoulders of the people that come before you. Did Newton patent calculus and screw the hell out of the other guy working on it? It's just so wrong, but it's what the business world does to things, and certainly the world is controlled by business interests. And technical idealists are a minority.

Electric Playground interviews John Carmack, 1997

Carmack got bitten by patents himself when he discovered a shadow drawing technique that somewhat ironically came to be known as Carmack's Reverse. While Carmack arrived at the technique independently, Creative had sought a patent for something very similar a few months earlier. The issue raised its head again this year with the Doom 3 source code release. In order to get legal approval to release the code, Carmack had to change the implementation to avoid infringing Creative's patent.

Patenting software really doesn't make sense. In programming there is usually an optimal and natural way of solving any given problem. By making optimal solutions exclusive, it reduces the efficiency with which people can create robust software. It's something that doesn't seem likely to encourage innovation to me. It seems much more likely to stifle it. And that's before we even consider the chilling effect that legal threats from gargantuan corporations can have on a would-be technologists.

Now you can manually do it yourself with apt-get purge , or, you can use the dpkg command to do it in a nice oneliner.

dpkg --get-selections | grep deinstall | sed 's/deinstall/\lpurge/' | dpkg --set-selections; dpkg -Pa

Start a VM

VBoxManage startvm "Name of VM" --type headless|gui

Forcibly Shutdown a VM

VBoxManage controlvm "Name of VM" poweroff

List VMs

VBoxManage list vms

Unregister/delete a VM

VBoxManage unregistervm "Name of VM" --delete

Forward / Unforward Ports for SSH

Modifyvm must be only be used when VM is powered off, otherwise controlvm for running VMs.

VBoxManage modifyvm "VM name" --natpf1 "guestssh,tcp,,2222,,22"
VBoxManage modifyvm "VM name" --natpf1 delete "guestssh"

$ skype
> skype: error while loading shared libraries: libXss.so.1: cannot open shared object file: No such file or directory

Hmmm.

$ locate libXss.so.1
> /usr/lib/x86_64-linux-gnu/libXss.so.1
> /usr/lib/x86_64-linux-gnu/libXss.so.1.0.0

Okay, so it's moaning about the X screensaver library, libxss, not being there. But it is there, although specifically, it's a 64bit library. I bet Skype isn't 64bit...

$ file /usr/bin/skype
> /usr/bin/skype: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, stripped

So that explains that then, we need compatible shared x86 libraries, but when you get these sorts of issues it's always best to see what else is missing. The ldd tool comes to the rescue.

$ ldd /usr/bin/skype
> linux-gate.so.1 =>  (0xf76f2000)
> libasound.so.2 => /usr/lib32/libasound.so.2 (0xf75df000)
> libXv.so.1 => /usr/lib32/libXv.so.1 (0xf75d9000)
> libXss.so.1 => not found
> librt.so.1 => /lib32/librt.so.1 (0xf75d0000)
> libQtDBus.so.4 => not found
> libQtGui.so.4 => not found
> libQtNetwork.so.4 => not found
> libQtCore.so.4 => not found
> libpthread.so.0 => /lib32/libpthread.so.0 (0xf75b4000)
> libstdc++.so.6 => /usr/lib32/libstdc++.so.6 (0xf74c9000)
> libm.so.6 => /lib32/libm.so.6 (0xf749e000)
> libgcc_s.so.1 => /usr/lib32/libgcc_s.so.1 (0xf7480000)
> libc.so.6 => /lib32/libc.so.6 (0xf7306000)
> libdl.so.2 => /lib32/libdl.so.2 (0xf7301000)
> libX11.so.6 => /usr/lib32/libX11.so.6 (0xf71cb000)
> libXext.so.6 => /usr/lib32/libXext.so.6 (0xf71b8000)
> /lib/ld-linux.so.2 (0xf76f3000)
> libxcb.so.1 => /usr/lib32/libxcb.so.1 (0xf7198000)
> libXau.so.6 => /usr/lib32/libXau.so.6 (0xf7194000)
> libXdmcp.so.6 => /usr/lib32/libXdmcp.so.6 (0xf718d000)

So looking at this, I do not have both compatible libxss and misc qt libraries installed.

With Ubuntu, to enable multiarch support, check the file /etc/dpkg/dpkg.cfg.d/multiarch. There should be a line like below (and if there isn't, add it):

# in file /etc/dpkg/dpkg.cfg.d/multiarch
foreign-architecture i386

Now we need to satisfy Skype's i386 dependencies.

 $ sudo apt-get install libxss1:i386 libqtcore4:i386 libqt4-dbus:i386 libqtgui4:i386

If we re-run ldd we can see Skype's shared library dependencies are now satisfied.

$ ldd /usr/bin/skype
> linux-gate.so.1 =>  (0xf7749000)
> libasound.so.2 => /usr/lib32/libasound.so.2 (0xf7634000)
> libXv.so.1 => /usr/lib32/libXv.so.1 (0xf762e000)
> libXss.so.1 => /usr/lib/i386-linux-gnu/libXss.so.1 (0xf7629000)
> librt.so.1 => /lib32/librt.so.1 (0xf7620000)
> libQtDBus.so.4 => /usr/lib/i386-linux-gnu/libQtDBus.so.4 (0xf75a6000)
> libQtGui.so.4 => /usr/lib/i386-linux-gnu/libQtGui.so.4 (0xf6ae0000)
> libQtNetwork.so.4 => /usr/lib/i386-linux-gnu/libQtNetwork.so.4 (0xf69a4000)
> libQtCore.so.4 => /usr/lib/i386-linux-gnu/libQtCore.so.4 (0xf6702000)
> libpthread.so.0 => /lib32/libpthread.so.0 (0xf66e7000)
> libstdc++.so.6 => /usr/lib32/libstdc++.so.6 (0xf65fc000)
> libm.so.6 => /lib32/libm.so.6 (0xf65d2000)
> libgcc_s.so.1 => /usr/lib32/libgcc_s.so.1 (0xf65b4000)
> libc.so.6 => /lib32/libc.so.6 (0xf6439000)
> libdl.so.2 => /lib32/libdl.so.2 (0xf6434000)
> libX11.so.6 => /usr/lib32/libX11.so.6 (0xf62fe000)
> libXext.so.6 => /usr/lib32/libXext.so.6 (0xf62eb000)
> libQtXml.so.4 => /usr/lib/i386-linux-gnu/libQtXml.so.4 (0xf62aa000)
> libdbus-1.so.3 => /lib32/libdbus-1.so.3 (0xf6261000)
> libfontconfig.so.1 => /usr/lib32/libfontconfig.so.1 (0xf622b000)
> libaudio.so.2 => /usr/lib32/libaudio.so.2 (0xf6211000)
> libglib-2.0.so.0 => /lib32/libglib-2.0.so.0 (0xf6118000)
> libpng12.so.0 => /lib32/libpng12.so.0 (0xf60ee000)
> libz.so.1 => /usr/lib32/libz.so.1 (0xf60d9000)
> libfreetype.so.6 => /usr/lib32/libfreetype.so.6 (0xf6041000)
> libgobject-2.0.so.0 => /usr/lib32/libgobject-2.0.so.0 (0xf5ff2000)
> libSM.so.6 => /usr/lib32/libSM.so.6 (0xf5fe9000)
> libICE.so.6 => /usr/lib32/libICE.so.6 (0xf5fcf000)
> libXi.so.6 => /usr/lib32/libXi.so.6 (0xf5fbf000)
> libXrender.so.1 => /usr/lib32/libXrender.so.1 (0xf5fb4000)
> libgthread-2.0.so.0 => /usr/lib32/libgthread-2.0.so.0 (0xf5fad000)
> /lib/ld-linux.so.2 (0xf774a000)
> libxcb.so.1 => /usr/lib32/libxcb.so.1 (0xf5f8e000)
> libexpat.so.1 => /lib32/libexpat.so.1 (0xf5f64000)
> libXt.so.6 => /usr/lib32/libXt.so.6 (0xf5f08000)
> libXau.so.6 => /usr/lib32/libXau.so.6 (0xf5f04000)
> libpcre.so.3 => /lib32/libpcre.so.3 (0xf5ec4000)
> libffi.so.6 => /usr/lib32/libffi.so.6 (0xf5ebd000)
> libuuid.so.1 => /lib32/libuuid.so.1 (0xf5eb7000)
> libXdmcp.so.6 => /usr/lib32/libXdmcp.so.6 (0xf5eb0000)

Skype now starts up and behaves as I would expect.

$ gcc -m32 -o test32 test.c

After a new install of Ubuntu 11.10 I was left scratching my head for a little bit with the following error:

> In file included from /usr/include/stdio.h:28:0,
>             from test.c:1:
> /usr/include/features.h:323:26: fatal error: bits/predefs.h: No such file or directory
> compilation terminated.

In order to build 32bit executables you need to install the i386 libc dev package.

$ sudo apt-get install libc6-dev-i386

For a little bit of 'fun' if you pass the -S -masm=intel arguments to gcc, you can compile once with -m32, and once with -m64 to see how your program changes based on target architecture. It can (perhaps it's just me :)) be Interesting to see the subtle changes in programs based on how they are assembled for x86_64 vs x86.

$ gcc -m32 -S -masm=intel test.c

Keeping up with changes to PHPUnit's suite of extensions can be occasionally dizzying, things have been put in and then ripped out of the core quite frequently in the last few releases. A common problem I'm seeing lately upgrading my test-suites to be 3.6 compatible, is the extraction of database testing classes to their own package on the pear.phpunit.de channel.

If you see an error like this:

> include(PHPUnit/Extensions/Database/TestCase.php): failed to open stream: No such file or directory

Odds are you need to do this:

$ sudo pear config-set auto_discover 1
$ sudo pear install --alldeps pear.phpunit.de/DbUnit

On a setup I administer, there are over 200,000 address records. When you view an order in the backend, and click 'edit address', the server grinds away and eventually dies either because it hits the max_execution_time limit or runs out of RAM.

You might see an otherwise meaningless error like this:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 79 bytes) in /home/somewhere/public_html/lib/Zend/Db/Statement/Pdo.php on line 290

The cause for this, is the strange manner in which Magento searches for addresses in app/code/core/Mage/Adminhtml/controllers/Sales/OrderController.

/**
 * Edit order address form
 */
public function addressAction()
{
    $addressId = $this->getRequest()->getParam('address_id');
    $address = Mage::getModel('sales/order_address')
        ->getCollection()
        ->getItemById($addressId);
    if ($address) {
        Mage::register('order_address', $address);
        $this->loadLayout();
        $this->renderLayout();
    } else {
        $this->_redirect('*/*/');
    }
}

The problem is in the $address->getCollection()->getItemById() chain. Magento creates and fully loads a collection of Address objects (which when you have 200k of them, takes a while). The final call in the chain, getItemById() takes the collection, iterates over it assigning each address to an array keyed by its entityId. It then returns any value which matches $addressId.

Now, there's another, really simple way to do the same thing. It doesn't involve instantiating 200,000 address objects, or iterating over them, or even using associative arrays. It's very familiar.

$address = Mage::getModel('sales/order_address')->load($addressId);

This one line does the same thing far more efficiently. Now, the thing that worries me, is I can't see any reason why they aren't doing this already? The change in code works, nothing appears to break and the speed boost is (obviously) immense.

So why is it not done this way?

$ varnishd -C -f /path/to/mysetup.vcl
> ...

Varnish will compile the file and print out it's full configuration as output. If there's an error, varnish handily gives you a compiler error to help trace what went wrong, which could look something like this

> Message from VCC-compiler:
> Expected an action, 'if', '{' or '}'
> ('input' Line 82 Pos 6)
>     vcl_hash(req.http.Cookie);
> -----########------------------
>
> Running VCC-compiler failed, exit 1

Some things still grate, particularly the global menus and the overlay scrollbar, but they are manageable.

A great resource I think to checkout for anyone using Oneiric is webupd8. You can find a lot of tips to bend Oneiric to your will.

For me, that means getting an official JVM installed Installing Oracle Java 7 and getting a sane desktop environment configured with global menu and overlay scrollbars zapped away Things to Tweak After Installing Oneiric.

Linux Mint (now on release 12, Lisa) is increasingly becoming a viable alternative to Ubuntu. I've played around with it a fair bit, but I still think it is not quite there yet in terms of 'just working' when compared with Ubuntu.

For someone that needs to get work done with a minimum of fuss, Ubuntu is still the leader. I do hope though, someone can work out how to marry usability and power in a slightly better way than I feel Canonical are managing at the moment.

Anyway, unlike either Enlightenment or E-Term, Imagemagick has been a firm friend ever since.

There are three main tools I find I fall back on, time and time again.

Display, Convert, and Identify.

Display, funnily enough, will open up an x-window with the contents of an image.

Convert, perhaps the most useful tool of them all, lets you resize, transform, and re-format images.

And lastly, identify, lets you get information about a file, size, format, colour depth, offset and so forth.

The power of these tools is best wielded with simple loops in bash, or using GNU Find. For example to convert a bunch of images from 1920x1080 to say 1280x720, you could do something like this

$ for IMAGE in *.jpg; do
>   # usage convert <action> orgfile.jpg newfile.jpg
>   convert -resize '1280x720' $IMAGE $(echo $IMAGE | sed 's/.jpg$/-resized.jpg/')
> done

Convert tries to be clever and maintain aspect ratios. In this case, 1920x1080 fits into 1280x720 maintaning a 16:9 aspect ratio. But what if we had a 1920x1200 input image? In these cases, convert would actually resize the image to something like 1280x800. But if we really want it to ignore common sense and squish things down to our requested 1280x720 you need to use the bang operator e.g.

$ convert -resize '1280x720!' srcimg.jpg dstimg.jpg

$ git push origin :REMOTE_BRANCH_NAME

For a concrete example of this in action

$ git branch -r
> origin/develop
> origin/master

$ git push origin :develop
> To /home/aaron/development/atestrepo.git
  - [deleted]         develop

$ git branch -r
> origin/master

So, why does this work, and why is the syntax so odd?

Well if we look at the push command in more detail it starts to make some sense.

If we want to push a local branch to a remote, we would do it like this

$ git push origin mybranch

Here, origin is the remote, and mybranch the local branch to push up. The result of this command is we now have a remote branch origin/mybranch. But what if we wanted to call the remote branch something else? We would do that like this:

$ git push origin mybranch:adiffnamefortheremotebranch

This syntax effectively translates to push to a branch adiffnameforremotebranch at remote origin the contents of mybranch. Now, can you see where this is going with respect to our delete? Deleting a remote branch with just a leading : (and no local branch name) is basically saying push nothing into remote branch called someremotebranch. Git takes this to mean delete the remote branch.

$ git push origin :someremotebranch

I find thinking of it in these terms, makes it easier to remember the syntax.

It's also worth reading the Pro Git Chapter on remotes which also covers (although, sadly, too briefly) deleting remote branches.

This is very, very handy if you want to do in-place substitutions with something like sed or in this case, the GNU text encoding conversion utility iconv.

Typically to convert between two encodings, you call iconv like this:

$ iconv -f cp1252 -t utf-8 myfile.txt

This will convert myfile.txt from windows latin 1, to utf-8 and dump the results to stdout.(Don't make the mistake of thinking iso-8859-1 and cp1252 are equivalent. It is safe to convert iso-8859-1 content as if it's cp1252, but the reverse is NOT true).

If you're in any doubt as to the encoding of the source file, you can inspect it with the 'file' command.

So if we have a directory of say c source files we want to convert, we can make use of bash, iconv and sponge to save us the tedium of converting each file manually to a new copy, then replacing the original file with the copy.

$ for FILE in *.c *.h; do iconv -f cp1252 -t utf-8 "$FILE" | sponge "$FILE"

Each file is filtered through iconv, the outputs of which are piped into sponge. Sponge soaks up the standard input until the EOF, then writes it to the original file.

A simple thing today I tried out today, downloading and extracting a web-based tarfile in a one liner.

It's really simple.

$ tar zxv < <(wget -q -O - http://www.somewhere.com/atarfile.tar.gz)
> atarfile/
> atarfile/file.txt
> ...

So let's look at this statement in a little detail. Basically we evaluate the right side of the expression first. We're using wget to download, quietly (-q), a tgz file from somewhere on the internet and output the file's contents to stdout (-O -).

We use Bash's Process Substitution operator '<(' to create a Named Pipe, which effectively creates a temporary file descriptor, and then direct the contents of that descriptor into tar using the '<' file redirection operator.

Sounds complicated but looks simple.

The UI for disabling modules is not nice, particularly once module dependencies are thrown into the mix. It's not a case of simply boring through the list of modules unchecking everything. First you need to uncheck everything you can. Save changes. Then go through the list again disabling the previously greyed out modules (because they still had active dependants) and disabling those. It can take about 4-5 passes before you're done.

Thankfully there is a much easier way using Drush. Drush is a command-line utility for managing aspects of a Drupal installation. Pertinently for Drupal upgraders it provides the ability to:

• List installed modules
• Disable modules

Running Drush without any arguments provides a list of commands and options. The two commands we are interested in are 'pm-list' (list installed modules) and 'pm-disable' (disable a module).

For these commands to work, you must issue your calls to drush from within your Drupal site's install directory e.g. /home/user/public_html/drupal

Keep in mind you can get information on any of Drush's commands by issuing a call to the help command.

$ drush help pm-list
> Show a list of available extensions (modules and themes).
> 
> Options:
>  --type                                    Filter by extension type. Choices:   
>                                            module, theme.                       
>  --status                                  Filter by extension status. Choices: 
>                                            enabled, disable and/or 'not         
>                                            installed'. You can use multiple     
>                                            comma separated values. (i.e.        
>                                            --status="disabled,not installed").  
>  --package                                 Filter by project packages. You can  
>                                            use multiple comma separated values. 
>                                            (i.e. --package="Core -              
>                                            required,Other").                    
>  --core                                    Filter out extensions that are not   
>                                            in drupal core.                      
>  --no-core                                 Filter out extensions that are       
>                                            provided by drupal core.             
>  --pipe                                    Returns a space delimited list of    
>                                            the names of the resulting           
>                                            extensions.                          
> 
> 
> Aliases: pml

We want a list of non-core modules and we would prefer non-formatted output so our command call looks like this:

$ drush pm-list --type=module --no-core --pipe
> ad
> ad_channel
> click_filter
> ad_embed
> ad_cache_file
> ad_notify
> ad_owners
> ad_report
> ...

Now can produce a list of non-core modules to pass to the disable module Drush command.

$ drush help pm-disable
> Disable one or more extensions (modules or themes). Disable dependant extensions as well.
>
> Arguments:
>  extensions                                A list of modules or themes. You can 
>                                           use the * wildcard at the end of     
>                                           extension names to disable multiple  
>                                           matches.                             
>
>
> Aliases: dis

Simple then. Putting it all together we can chain together the module list command output using shell command substitution and pass that output as arguments to the disable command, all in one neat line.

$ drush pm-disable `drush pm-list --no-core --type=module --pipe`

Press 'y' in response to the resulting confirmation prompt and the tedious work in the Drupal admin UI is replaced with a neat one-line shell command!

Branching and merging on a local repository is trivially simple, but what if you create a local branch on a repository that otherwise syncs up with a remote repository?

Follow these steps (this assumes a remote called origin is set up)

$ git branch my-new-feature
$ ... # hack away
$ git commit -a -m 'Initial feature commit'
$ git push origin my-new-feature
$ git branch --set-upstream my-new-feature origin/my-new-feature

Git pull/fetch/push will now track the newly created remote branch called 'my-new-feature'.

$ usermod -a -G wheel aaron

The -a argument is very important, it ensures arguments to -G append to the existing list of groups. Otherwise existing groups will be replaced with the argument supplied.

With an application I am currently configuring for Jenkins, it was taking about 30-35 minutes to generate all the API documentation through PHPDocumentor. I had heard good things about DocBlox - must notably its speed - and it's used by the Zend Framework so I thought I'd sub it in for PHPDocumentor and see if it was quicker.

Installation

DocBlox has a number of dependencies which we also need to install if they are not already.

• XSL PHP
• PHP-Markdown
• Graphviz

Use your system's package manager to install the xsl php extension and graphviz.

$ sudo aptitude install php5-xsl
$ sudo aptitude install graphviz

To install DocBlox and PHP-Markdown use PEAR.

$ sudo pear channel-discover pear.michelf.com
$ sudo pear channel-discover pear.docblox-project.org
$ sudo pear install --alldeps channel://pear.docblox-project.org/DocBlox-0.13.1

How Fast?

DocBlox uses largely the same command line options as phpdoc, so it can act as a drop in replacement if you already have existing build scripts utilising phpdoc.

In terms of speed, well, it's very fast. That 35 minute phpdoc run took just 43 seconds in DocBlox.

Git stash to the rescue:

$ git stash save
$ git checkout develop
$ git stash apply
$ git commit -m 'Apply stashed changes'

To start developing on this branch instead of the master, use the following command

$ git checkout -b develop origin/develop
> Branch develop set up to track remote branch develop from origin.
> Switched to a new branch 'develop'

This command creates a local branch tracking the remote develop branch and switches your working directory to this branch.

If you already have a local develop branch and want it to track the remote you can use this instead

$ git checkout develop
$ git branch --set-upstream develop origin/develop

$ sudo pear channel-discover pear.zfcampus.org
$ sudo pear install zfcampus/ZF

When configuring applications, if you wish to use an application's bundled copy of ZF, ensure local libs come first in PHP's include_path.

To solve the issue change into your jenkins home dir (/var/lib/jenkins on Ubuntu) and add the following to a file named .gitconfig.

[user]
    name = Jenkins
    email = jenkins@localhost

That will allow you to avoid this:

Started by user anonymous Checkout:workspace / /var/lib/jenkins/jobs/Bookings/workspace - hudson.remoting.LocalChannel@289d9155 Using strategy: Default Checkout:workspace / /var/lib/jenkins/jobs/Bookings/workspace - hudson.remoting.LocalChannel@289d9155 Cloning the remote Git repository Cloning repository origin Fetching upstream changes from file:///home/aaron/Sites/Bookings Seen branch in repository origin/HEAD Seen branch in repository origin/master Commencing build of Revision fc8a70bc628edb0ac547b3fefc0841db5e06204c (origin/HEAD, origin/master) Checking out Revision fc8a70bc628edb0ac547b3fefc0841db5e06204c (origin/HEAD, origin/master) FATAL: Could not apply tag jenkins-Bookings-1 hudson.plugins.git.GitException: Could not apply tag jenkins-Bookings-1 at hudson.plugins.git.GitAPI.tag(GitAPI.java:698) at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1181) at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1129) at hudson.FilePath.act(FilePath.java:758) at hudson.FilePath.act(FilePath.java:740) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1129) at hudson.model.AbstractProject.checkout(AbstractProject.java:1193) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:555) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:443) at hudson.model.Run.run(Run.java:1376) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:175) Caused by: hudson.plugins.git.GitException: Error performing command: git tag -a -f -m Jenkins Build #1 jenkins-Bookings-1 Command "git tag -a -f -m Jenkins Build #1 jenkins-Bookings-1" returned status code 128: *** Please tell me who you are.

Run

git config --global user.email "you@example.com" git config --global user.name "Your Name"

to set your account's default identity. Omit --global to set the identity only in this repository.

fatal: empty ident <jenkins@blizzard.(none)> not allowed

at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:744)
at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:709)
at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:719)
at hudson.plugins.git.GitAPI.tag(GitAPI.java:696)
... 12 more

Caused by: hudson.plugins.git.GitException: Command "git tag -a -f -m Jenkins Build #1 jenkins-Bookings-1" returned status code 128: *** Please tell me who you are.

If you do not have Jenkins installed or are not sure you have your environment setup correctly, please checkout my previous tutorial Getting Started With Jenkins for PHP.

A quick checklist

Before we start, make sure you have your environment and Jenkins installation up and running and appropriately configured.

• You have a JVM and Jenkins installed running at http://localhost:8080
• You have the following Jenkins plugins installed
  • checkstyle
  • cloverphp
  • dry
  • htmlpublisher
  • jdepend
  • plot
  • pmd
  • violations
  • xunit
• You have the following PEAR packages installed
  • pdepend/PHP_Depend
  • phpmd/PHP_PMD
  • phpunit/phpcpd
  • phpunit/phploc
  • PHPDocumentor
  • PHP_CodeSniffer
  • phpunit/PHP_CodeBrowser
  • phpunit/PHPUnit
  • phpunit/ppw
• You have Ant installed
• You have Git installed and are familiar with its basic use
• You have the Zend_Framework installed globally as a PEAR package

If you don't have the above installed please refer to the Getting Started With Jenkins for PHP tutorial.

If you're not familar with Git, have a quick read of git-scm.org or refer to my Git cheatsheet.

To install the Zend Framework as a PEAR package see my post on Installing Zend Framework from PEAR.

Create your project

I'm basing this tutorial on a sample Zend Freamework application, you're free to use whatever codebase you like. But for the purposes of this tutorial I'm using this project as a starting point: git@github.com:ajbonner/Bookings.git.

$ git clone https://github.com/ajbonner/Bookings.git ./bookings
$ cd bookings
$ ppw --name 'Bookings' --source ./application .

This will generate an ant build file with some general defaults for PHP_CodeSniffer and PHP Mess Detector along with a standard phpunit configuration file.

First thing, we'll need to make a small amendment to the phpunit configuration file, since we're testing a Zend Framework application, we need to make sure PHPUnit loads a custom bootstrap before running tests. Open up the generated phpunit.xml.dist file and change the opening phpunit element to look like this:

<phpunit backupGlobals="false"
              backupStaticAttributes="false"
              strict="true"
              verbose="true"
              colors="true"
              bootstrap="tests/bootstrap.php">

Our build environment is now configured correctly to run Zend Framework Test Cases, we have our ant build file setup and some defaults configured for our code coverage and analysis tools. Let's add our new build and phpunit configuration files to git and commit our changes. Then we can run ant and see what happens.

$ git add build.xml phpunit.xml.dist
$ git commit -m 'Add build and test configuration files'
$ ant
>> Buildfile: /home/aaron/Sites/Bookings/build.xml
>> clean:
>>    [delete] Deleting directory /home/aaron/Sites/Bookings/build/api
>>    [delete] Deleting directory /home/aaron/Sites/Bookings/build/code-browser
>>    [delete] Deleting directory /home/aaron/Sites/Bookings/build/coverage
>>    [delete] Deleting directory /home/aaron/Sites/Bookings/build/logs
>>    [delete] Deleting directory /home/aaron/Sites/Bookings/build/pdepend
>>     [mkdir] Created dir: /home/aaron/Sites/Bookings/build/api
>>     [mkdir] Created dir: /home/aaron/Sites/Bookings/build/code-browser
>>     [mkdir] Created dir: /home/aaron/Sites/Bookings/build/coverage
>>     [mkdir] Created dir: /home/aaron/Sites/Bookings/build/logs
>>     [mkdir] Created dir: /home/aaron/Sites/Bookings/build/pdepend
>> 
>> parallelTasks:
>> 
>> phpcpd:
>> 
>> pdepend:
>>      [exec] PHP_Depend 0.10.5 by Manuel Pichler
>>      [exec] 
>>      [exec] Parsing source files:
>>      [exec] ....                                                             4
>>      [exec] 
>>      [exec] Executing Coupling-Analyzer:
>>      [exec]                                                                 18
>>      [exec] 
>>      [exec] Executing CyclomaticComplexity-Analyzer:
>>      [exec] .                                                               24
>>      [exec] 
>>      [exec] Executing Dependency-Analyzer:
>>      [exec]                                                                 17
>>      [exec] 
>>      [exec] Executing Inheritance-Analyzer:
>>      [exec]                                                                  5
>>      [exec] 
>>      [exec] Executing NodeCount-Analyzer:
>>      [exec]                                                                 14
>>      [exec] 
>>      [exec] Executing NodeLoc-Analyzer:
>>      [exec] phpcpd 1.3.2 by Sebastian Bergmann.
>>      [exec] 
>>      [exec] 0.00% duplicated lines out of 119 total lines of code.
>>      [exec] 
>>      [exec] Time: 0 seconds, Memory: 2.75Mb
>>      [exec] .                                                               22
>>      [exec] 
>>      [exec] Generating pdepend log files, this may take a moment.
>>      [exec] 
>>      [exec] Time: 00:00; Memory: 9.50Mb
>> 
>> phpcs:
>> 
>> phpmd:
>>      [exec] Result: 1
>> 
>> phploc:
>>      [exec] phploc 1.6.1 by Sebastian Bergmann.
>>      [exec] 
>>      [exec] Directories:                                          1
>>      [exec] Files:                                                4
>>      [exec] 
>>      [exec] Lines of Code (LOC):                                119
>>      [exec]   Cyclomatic Complexity / Lines of Code:           0.08
>>      [exec] Comment Lines of Code (CLOC):                        18
>>      [exec] Non-Comment Lines of Code (NCLOC):                  101
>>      [exec] 
>>      [exec] Namespaces:                                           0
>>      [exec] Interfaces:                                           0
>>      [exec] Classes:                                              4
>>      [exec]   Abstract:                                           0 (0.00%)
>>      [exec]   Concrete:                                           4 (100.00%)
>>      [exec]   Average Class Length (NCLOC):                      22
>>      [exec] Methods:                                              9
>>      [exec]   Scope:
>>      [exec]     Non-Static:                                       9 (100.00%)
>>      [exec]     Static:                                           0 (0.00%)
>>      [exec]   Visibility:
>>      [exec]     Public:                                           8 (88.89%)
>>      [exec]     Non-Public:                                       1 (11.11%)
>>      [exec]   Average Method Length (NCLOC):                     10
>>      [exec]   Cyclomatic Complexity / Number of Methods:       1.89
>>      [exec] 
>>      [exec] Anonymous Functions:                                  0
>>      [exec] Functions:                                            0
>>      [exec] 
>>      [exec] Constants:                                            0
>>      [exec]   Global constants:                                   0
>>      [exec]   Class constants:                                    0
>> 
>> phpunit:
>>      [exec] PHPUnit 3.5.14 by Sebastian Bergmann.
>>      [exec] 
>>      [exec] ...
>>      [exec] .
>>      [exec] 
>>      [exec] Time: 1 second, Memory: 19.25Mb
>>      [exec] 
>>             OK (4 tests, 16 assertions)
>> 
>>      [exec] Writing code coverage data to XML file, this may take a moment.
>>      [exec] 
>>      [exec] Generating code coverage report, this may take a moment.
>> 
>> phpcb:
>> 
>> build:
>> 
>> BUILD SUCCESSFUL
>> Total time: 3 seconds
>> 

If you have everything setup correctly you will have a build directory with a number of artifacts: code coverage analysis, code style reports, dependency analysis, etc. You can look at these directly but it's much more convienient to use Jenkins to view these artifacts.

Importing a Project into Jenkins

We have a working build and we are producing a number of artifacts representing certain aspects of our software. Time now to import the project into Jenkins and have this process ran automatically.

Step 1 - Create a New Jenkins Project

As part of getting our environment ready for Jenkins, we installed a PHP Project Template. To create our new Project, we copy this template, and give our project a name. Choose whatever you like, but for the purposes of this tutorial, I've chosen 'Bookings'.

Step 2 - Configure your New Project

You're now presented with a (at first glance) daunting configuration page. We'll work our way down the page, for the most part we don't need to make major changes.

Firstly the pdepend task generates a pair of SVG images, in order to have these display on the project dashboard we need to substitute 'job-name' to 'Bookings' (or whatever you chose for your project name) in the embed tags in the description field.

Next, uncheck 'disable build', naturally we want to perform builds!

The project will be retrieved from the git repository we created during the initial build configuration. Pass in the absolute filesystem path to your repository, for example mine is file:///home/aaron/Sites/Bookings.

That's it. You now have setup the project in Jenkins and defined a source. The remaining details can be left at their defaults. Scroll down to the bottom of the page and click the save button.

Step 3 - Build the Project

The project needs to first do a build before the project workspace can be initialised. On the left hand project menu is a 'build now' link. Click that and Jenkins will checkout a copy of our project from git then process the ant build.xml we defined.

Once the build is complete you can click on the build and take in all the dianostic and code analysis artifacts generated during the build.

Possibly, you may run into an issue building your project from git. A common cause is Jenkins being unable to find a user identity for its operating system user. Please see my post Cannot Build Git Project in Jenkins if this happens to you.

What Next?

Continuous integration isn't about pretty graphs and pithy build summaries. It's about maintaining working software and highlighting issues as soon as possible. Jenkins has a wealth of plugins that help this process, but perhaps the first one to checkout is the email functionality, letting your development team know immediately if something is broken.

Assuming you're running your server on port 8080, obtain the cli jar file like this:

$ wget http://localhost:8080/jnlpJars/jenkins-cli.jar

To see a list of available commands simply run the tool with the help argument:

$ java -jar jenkins-cli.jar -s http://localhost:8080 help
> build
>   Builds a job, and optionally waits until its completion.
> clear-queue
>   Clears the build queue
> connect-node
>   Reconnect to a node
> copy-job
>   Copies a job
> ...

In recent versions of Jenkins, typical server operations have been decoupled from the cli tool and are now issued using simple http requests. For example to reload Jenkins instance's configuration, you would just fire a http request at it like this:

$ curl http://localhost:8080/reload

There are three commands of this sort:

• reload Reload server configuration
• restart Restart the server
• exit Close the server down

Issue these in the format:

$ curl http://[jenkins-server]/[command]

There's two options, you can download a pre-prepped specific package and install using aptitude or synaptic, or you can set-up an update site to ensure you're always notified of updates.

I prefer the latter and used the following steps:

$ wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
$ sudo echo 'deb http://pkg.jenkins-ci.org/debian binary/' >> /etc/apt/sources.list
$ sudo apt-get update
$ sudo apt-get install jenkins

By default, the Jenkins packages attempt to start up a java web container running on port 8080. To change this, open up /etc/default/jenkins and look for the following:

# port for HTTP connector (default 8080; disable with -1)
HTTP_PORT=8080

Change the port number to whatever works for you and (re)start the service

$ sudo service jenkins restart

magento-quickies:

When you’re creating custom Magento models, it’s common to to add and remove fields to and from a database table during development before finalizing a set of columns for the setup resource/migration file. One thing to keep in mind while you’re doing this is that Magento will cache the list of table columns each model knows about, which means a cache clearing will be needed after adding a column.

Another great thing it can do, is resume broken file transfers e.g. from a sftp, scp, http or ftp transfer.

rsync --partial --progress --rsh=ssh localfile remoteuser@remotehost.com:/remote/path

There is some trace though, a hs_* dump file in your home directory. In that file is a message (along with a lot of other verbiage) like this

# Problematic frame:
# C [libswt-pi-gtk-3659.so+0x3c17e]
Java_org_eclipse_swt_internal_gtk_OS_GTK_1ACCEL_1LABEL_1GET_1ACCEL_1STRING+0x0

Not really helpful but it helped google lead me to a solution. That solution is here: https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/789123 (see comment 18).

To summarise, you basically need to uninstall overlay-scrollbar and liboverlay-scrollbar-0.1-0.

This can be achieved either using the package manager or aptitude

sudo aptitude remove overlay-scrollbar liboverlay-scrollbar-0.1-0

This has the added benefit of removing the annoying scrollbars that come with Natty.

Turns out with a little

gem install image_size

It's pretty easy!

Personally, I maintain a fairly vanilla setup. I use the PHP Developer Tools, E/JGIt plugins and Subclipse for SVN.

To get hold of these packages add the following two update sites after installing indigo.

• http://download.eclipse.org/releases/indigo - Egit and PDT 3
• http://subclipse.tigris.org/update_1.6.x - Subclipse SVN (much better than Subversive)

Turns out there's already something there to do it

Mage_Payment_Method_Abstract::_debug($data);

If you want to call it from outside the payment_method inheritance tree use

Mage_Payment_Method_Abstract::debugData($data);

In both cases your payment method needs to have its debug config setting enabled e.g. for my usaepay module

echo Mage::getStoreConfig('payment/usaepay/debug'); 
>> 1

This is where the git remote add command comes in handy.

$ git remote add origin git@github.com:ajbonner/foo.git

You can see the remotes associated with your branch:

$ git remote -v
>> origin	git@github.com:ajbonner/foo.git (fetch)
>> origin	git@github.com:ajbonner/foo.git (push)

To push the existing code to the remote:

$ git push origin master

You now have set up the remote and pushed your master branch into it. From here it gets tricky because subsequent git pull requests will give you an ugly error

$ git pull
>> You asked me to pull without telling me which branch you
want to merge with, and 'branch.master.merge' in
your configuration file does not tell me, either. Please
specify which branch you want to use on the command line and
try again (e.g. 'git pull <repository> <refspec>').
See git-pull(1) for details.

If you often merge with the same branch, you may want to
use something like the following in your configuration file:

[branch "master"]
remote = <nickname>
merge = <remote-ref>

[remote "<nickname>"]
url = <url>
fetch = <refspec>

See git-config(1) for details.

There a few ways to get around this:

1. Follow the directions given by git and edit your gitconfig

2. Clone a fresh copy of the master branch from the remote

3. When defining the remote add the --track option and give it the name of the master branch

   $ git remote add --track master origin git@github.com:ajbonner/foo.git

4 Refer to the remote branch using --set-upstream

$ git branch --set-upstream master origin/master

Personally I find option number 4 the best with the least amount of work.

You can also use:

$ git config --global branch.autosetupmerge true

To avoid having to do this.

On my most recent project I exclusively employed Mage_Test for my Unit Testing and with a few exceptions, it's performed extremely well.

Mage_Test takes a very hands-off, light-weight approach. EcomDev meanwhile, on the surface, appears to provide far greater support for testing at the expense of some complexity.

Installing EcomDev's Test module is easy. You can install it from Magento Connect or you can get the module directly from the developer's Subversion repository

$ svn co http://svn.ecomdev.org/svn/ecomdev-phpunit/trunk

Once you have the code, it's just a matter of copying into your magento store as with any other module:

$ cp -r <ecomdevtestdir>/* path/to/your/magento/rootdir

Once the files are in place, you'll need to edit app/etc/local.xml.phpunit and supply some details for a test database connection and some path information relating to the store site root URIs.

Once your database and paths have been defined initialise the test database by changing into your store root and running the UnitTests.php test suite

~/Sites/my/store/root: $ phpunit UnitTests.php

This will take some time (one of my clients has a 400meg database and a lot of orders, it took ~ 4 minutes).

You're now ready to start writing some tests.

For more information on EcomDev's PHPUnit module see this blog post or get a copy of the (very) comprehensive EcomDev PHPUnit 0.20 Manual

mysqldump -uuser -p mydb mytables_*

There isn't an inbuilt mechanism to do this, my reply was to use a shell script with an array containing a list of tables you wanted to export. Another reply had a better way, using a call to mysql to get a list of tables matching a glob pattern, putting them in an array and then iterating over that array with successive calls to mysqldump.

I put this suggestion together with mine, made it more generic and popped it on github for reference.

Personally I've placed this in my user's .bash_functions (included by .bashrc).

There is an issue at the moment when you try and install the egit/jgit plugins with Indigo versions RC3 and above.

You'll encounter an error similar to this:

An error occurred while collecting items to be installed session context was:(profile=epp.package.rcp, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect, operand=, action=). No repository found containing: osgi.bundle,org.eclipse.egit,1.0.0.201106011211-rc3 No repository found containing: osgi.bundle,org.eclipse.egit.core,1.0.0.201106011211-rc3 No repository found containing: osgi.bundle,org.eclipse.egit.doc,1.0.0.201106011211-rc3 No repository found containing: osgi.bundle,org.eclipse.egit.ui,1.0.0.201106011211-rc3 No repository found containing: osgi.bundle,org.eclipse.jgit,1.0.0.201106011211-rc3

This is caused by the latest jgit packages not yet being in the indigo release p2 update repository. To get these installed until these updates are provided, add http://download.eclipse.org/egit/staging/ as an update site and you'll be able to install these plugins successfully.

I found this solution here: https://bugs.eclipse.org/bugs/show_bug.cgi?id=348183

$keys = array('mykey', 'another_key');
$array = array();
foreach ($keys AS $key) { $array[$key] = "hello world\n"; }
foreach ($keys AS $key) { echo $array[$key]; }

>> hello world
hello world

In Ruby, the presence (and use of Symbols) makes this a bit tricky:

keys = ['mykey', 'another_key']
myarray = { :mykey => 'hello world', 'another_key' => 'goodbye world' }
myarray.each { |k| 
  puts myarray[k]
}
 
>> hello world
goodbye world

Now if we try this again using instead the Strings from the keys array we get a different result:

keys.each { |k| 
  puts myarray[k]
}
 
>>
goodbye world

We don't get the first array value back because a String key is not the same as a Symbol key, even if they consist of the same sequence of characters. This is one of the gotchyas with Ruby.

In Ruby the 'value of a Symbol is not the same as that of a String. So :key != "key". From an ease of use perspective it would be convenient if they did.

Thankfully the language stewards saw fit to include String.to_sym as a convenience method to create a Symbol from a String's value.

keys.each { |k| 
  puts myarray[k.to_sym]
}
 
>> hello world

Pull down the 'classic' version Eclipse 3.7 Indigo from http://www.eclipse.org/downloads/ and install.

Once installed, launch Eclipse and navigate to Help->Install new Software.

Add the Indigo update site 'http://download.eclipse.org/releases/indigo'. This will take sometime to add, let it go for 5 or so minutes.

UPDATE 20/10/2011 The following step is no longer required as the the PDT 3.0 series is now in the main indigo update repository. Once the Indigo Update Site is added, add the PDT 3.0 Update Site http://download.eclipse.org/tools/pdt/updates/3.0/milestones/

Now, to install simply select PDT Development Tools All in One SDK (leave the others unselected) and click next. The installation process shouldn't take more than a few minutes. .

I had a feeling I should give them respect as much of the learning Ruby literature says to basically ignore them for now, or launch into lengthy defences of why they exist (usually forgetting to explain how they work).

In case you're unfamiliar with Ruby syntax, a Symbol looks like this

:a_symbol

With the numerous ways of defining variables in Ruby this is another variable looking constuct, except that it isn't actually a variable at all (you cannot assign to it).

So what is a Symbol, actually. To be honest I'm still struggling for a good simple definition. I think the best way to think of it, is as a constant (i.e. immutable) placeholder for the name of something. Or more practically put, a colon in front of the string you want to use for a hashkey. Agile Programming With Rails 4th Ed puts it simply 'a Symbol is a name for something'.

I wrote on Twitter that I felt Symbols were extra syntax put in to solve a problem a programmer in a high level interpreted language shouldn't have to worry about. Memory management. In Ruby everything is an object, and objects are much bigger than the primitives you have in Java, C or even PHP. A string for example in PHP (and as in C) is still ultimately a sequence of bytes stored in contiguous memory addresses. In Ruby, a string is always an object, and that requires a fair chunk of memory to represent. If you have a bunch of hash tables, using objects as a hashkey is inefficient and wasteful.

Symbols help solve this inefficiency. A Symbol is still an object, but a special one. It has few methods the main ones being to get its string value and another its integer value, it's immutable and there's only ever one copy of it. So it's much more efficient to use a symbol as a hashkey than a Ruby string. In Ruby (and most OO languges) two strings even if consisting of the same sequence of characters are different objects. In Ruby two symbols of the same sequence of characters are the same object. In large applications this feature can save a tremendous amout of memory.

The other key characteristic of Symbols is immutability. Symbols cannot be assigned to, they just are. In a language that lacks a true constant construct (uppercasing a variable name is the convention for defining a Ruby constant but read-only access isn't enforced at runtime, you can write to what Ruby calls a 'constant'), Symbols can be useful.

So are Symbols a good language feature? At this point with my lack of experience with Ruby I don't really feel qualified to answer that yet definitively. My gut though, says no. I feel in a high level language the need for extra syntax to optimise code adds an unnecessary burden on the programmer. They seem so out of place in a language that works so hard to strip away unneccessary syntax. Symbols, to me detract from Ruby's power to define elegant and natural sounding expressions.

The difficulty authors have in describing what Symbols are, how to use them and why they should be used seems like a language design smell to me. I dare say as I become more familiar and accustomed to the presence and use of Symbols I'll learn to accept them. But as a Developer new to Ruby, they seem out of place.

I haven't gone into an excessive amount of detail about the nature and application of Symbols. For that the two best resources I've found explaining are http://glu.ttono.us/articles/2005/08/19/understanding-ruby-symbols and http://www.troubleshooters.com/codecorn/ruby/symbols.htm.

The Basics

The process for creating and restoring a snapshot is trivial

$ mysqldump -uuser -p mydatabase > mydump.sql    # export
$ mysql -uuser -p mydatabase < mydump.sql        # import

For the most part this works as you expect it to, and for small databases this is probably all that is needed. You may find the resultant .sql file is huge, it is after all uncompressed text, at 1 byte per character if the output is ANSI or up to 4 bytes if your character set is UTF-8. Bzip2 will bring the file size down a considerable amount but it’s also considerably slower than gzip.

It’s sometimes tempting to gzip/bzip2 your datadump while performing the mysqldump in a single line.

$ mysqldump -uuser -p mydatabase | bzip2 -c > mydump.sql.bz2

While it seems a nice efficient way to do your backup, this should be avoided as (by default in MyISAM) you’re locking tables and denying other clients access to them. InnoDB implements row level locking which is slightly less offensive, but still should be avoided as much as possible.

When importing a large database, the choice of zip format is important. You have to trade off decompression speed, with filesize. The extra cputime consumed decompressing a bzip2 datadump may actually be less preferable to a few extra megabytes gained by using the faster gzip. Whatever your choice, importing a zipped datadump is very easy.

$ gunzip -c mydump.sql.gz | mysql -uuuser -p mydatabase  # importing a gzipped datadump
$ bunzip2 -c mydump.sql.bz2 | mysql -uuuser -p mydatabase # importing a bzip2 datadump

Locking and Transactional Integrity

As explained briefly above, MySQL’s storage engines come with some limitations. In the worst case, with MyISAM, while performing a mysqldump entire tables will be locked, that means other clients will not be permitted to write to a table while the dump is being performed. If you have large MyISAM tables this poses clear problems when backing up a running application.

InnoDB is slightly better because it uses row level locking. It locks only the rows affected by a query, and not the whole table. This makes a conflict far less likely to occur while performing a backup on a running application. InnoDB as a transactional storage engine does allow for the possibility that active transactions may be underway while you’re attempting your backup.

Two options we can pass to mysqldump mitigate these issues are, —single-transaction and —skip-lock-tables.

$ mysqldump --single-transaction --skip-lock-tables mysql -uuser -p mydatabase

The use of —single-transaction means mysql issues a begin statement before dumping the contents of a table, ensuring a consistent state of the table without blocking other applications. It means writes can occur while the backup is taking place and this will not affect the backup. The —skip-lock-tables option stops MyISAM tables being locked during the backup. This does mean the integrity of the table can be lost as writes occur to them during the backup process. The risk is weighed up against the risk of blocking access to the table during a lengthy backup process.

Improving Import Performance

Choice of zip format will have a large bearing on import performance. Gzip is appreciably faster than bzip2. Other options you can pass to mysqldump to improve import performance are —disable-keys and —no-autocommit.

Disabling keys significantly improves the performance of imports as mysql will only index the table at the end of the import. With keys enabled, the index is updated after each row is inserted. Given you are performing a batch import, this is suboptimal.

By default each statement in an InnoDB table is autocommitted. This comes with unneccessary overhead when performing a batch import as you really only need to commit once the table has been fully imported.

Further Reading

This only a brief look at using Mysqldump for backups. It’s a common enough development task that all developers should take the time to see how it can be best leveraged for their environment. There’s plenty of documentation out there on using the tool. But the best place to start is with the official docs.

"The quick brown fox", "The quicker brown fox".

If you try and use the VIM substitute regexp s/"The quick brown.*"// you'll end up nuking the whole line up to the terminating period mark. This is because the regular expression is acting greedily, and matching up to the second and final " character.

In PERL compatible regexps you would simply modify the expanding part of the regexp to be .*? which makes the expression match as little as possible.

In VIM the syntax is slightly different. To do a non-greedy expansion, you use .\{-}. For example:

s/"The quick brown.\{-}"//

This will leave you with 

, "The quicker brown fox".

$ git init <dirname>

Creates a new bare git repository in the current directory, or if a directory argument is given, in that directory.

Version control a file

$ git add file|dir <file|dir> ...

Adds, or to use git terminology ‘stages’ a file for a local commit.

Remove a version controlled file

$ git rm -rf --cached <file|dir>

This removes a file that has been previously staged (i.e added) for local commit. It does not remove the local file. If you omit the —cached option, the local file WILL be deleted. The rm command is roughly analagous to: $ svn del

Make a local commit

$ git commit <-a> -m 'Commit message' <file> ...

The -a flag will commit all pending files recursively in the current path, alternatively you can specify the exact files to commit at the end of the command separated by spaces.

See pending actions or status

$ git status <-u no|all|normal> <path> ...

Will show pending files and untracked files in the current working path. Optionally the -u (—untracked-files) option can be supplied to hide/show untracked files in the status report. Passing a path, or number of paths will run the status for that path rather than the current directory.

Create a bare centralised remote repository

$ git clone --bare pathtoinitialrepo <myrepo.git>

Alternatively if you don’t have an existing repo already:

$ git init --bare <path>

Often you’ll see the .git extension bandied about, particularly on github. This is a canonical reference to a bare git repository (i.e. only has the meta information and not a working copy) that is usually acts as a master or central repository.

Configuring a local repository to push to a remote

$ cd pathtoinitialrepo
$ git remote add origin ssh://myuser@somehost.com/~/repos/myrepo.git
$ git push origin master

If you initially clone from the remote repository you can skip the first step of setting up the origin. However if you have just created a new bare remote repository and want to setup an initial commit, you need to setup the origin first. Subsequent pushes to the remote repository can be done with git push origin master. This is the familiar model of subversion and svn commit.

Create a local copy of a remote git repository

$ git clone ssh://myuser@somehost.com/~/repos/myrepo.git

Create a local copy from remote repository created following the above approach.

Ignore changes to a tracked file

$ git update-index --assume-unchanged <file>

Use this command if you’ve added a file, e.g. a database defaults config file, but do not want further changes to be picked up for it.

To resume tracking changes to the file use:

$ git update-index --no-assume-unchanged <file>

I'm still very new to Ruby and its tooling so I was finding it hard to see the details of available remote gems.

I was labouring:

$ gem search --remote tumblr

*** REMOTE GEMS ***
integrity-tumblr (0.1.1)
matenia-tumblr-api (0.1.6)
ruby-tumblr (0.0.2)
skinnycms_tumblr (0.0.2)
tumblr (2.2.0)
tumblr-api (0.1.4)
tumblr-rb (1.3.0)
tumblr4r (0.7.2)
tumblr_cleanr (0.0.1)

Then for each interesting result: 

$ gem specification --remote <gem>

This brought back a lof interesting information but it is hardly an efficient way to browse. Digging into the docs I found an easier way.

$ gem search -r -d tumblr

*** REMOTE GEMS ***
integrity-tumblr (0.1.1)
    Author: Matías Flores
    Rubyforge: http://rubyforge.org/projects/integrity
    Homepage: http://github.com/matflores/integrity-tumblr
    Tumblr notifier for the Integrity continuous integration server
 
matenia-tumblr-api (0.1.6)
    Author: Jeff Kreeftmeijer, Matenia Rossides
    Homepage: http://github.com/matenia/tumblr
    Tumblr API wrapper - maintained by matenia

The -r is the short notation for --remote, and -d is the short notation for --details. The default for the search action is --no-details, which is why the initial resultset was so unhelpful.

$ sudo pear upgrade-all
> PHP Fatal error:  Call to undefined method PEAR::raiseErro() in /usr/share/php/PEAR/REST.php on line 165
PHP Stack trace:
...

You can fix this nuisance:

$ sudo mkdir -p /tmp/pear/cache
$ sudo pear upgrade-all

Hope this helps save someone some time!

$ gem server

Open your browser and go to 127.0.0.1:8808 and read all the api docs for your installed gems (i.e Rails and friends).

That, is very cool!

Typically a PHP project will start off small, some basic webpage views, maybe a few forms, and likely, some sort of search functionality. This is pretty basic and if things need to change, you can normally change it in place, directly on the web-server and without too much grief.

At some point though entropy takes its toll and incremental changes have so may unintended side-effects it's no longer feasible to safely make edits in-place. A quick hack directly on the web-server stops being a 'quick win', and more like a game of Russian Roulette.

This is where unit and integration testing comes to the fore. A safety net to protect us when we start changing an application. But often getting a test environment set-up and representative of the live system is a lot of work in itself, and the temptation to just give into the 'Inner Pig' is too great. That is, to not bother running any tests, and just cross fingers.

Jenkins makes this a lot easier by doing much of the heavy lifting of building and running tests. It will check that code complies with defined style conventions. It can also check for common coding smells (copy/paste, duplication, long methods, large classes, poor expressions) and it can run custom scripts depending on the success of unit and integration tests.

Essentially Jenkins ensures that any changes that go into source control, do not 'break the build' in a process known as Continuous Integration.

Installation is very easy. Visit the Jenkin's site, follow the instructions for your platform and you'll have a Jenkins's install running (by default) at http://localhost:8080. I have also written a small installation tutorial: Installing Jenkins on Ubuntu/Debian Systems.

To get Jenkins initially dressed up for PHP follow the jenkins-php.org site upto and including the setup the Pear packages and Jenkins plugins.

You may or may not run into trouble with these instructions. For the most part it worked fine for me, however I chose to use the cli-tool and the plugin repository was not initialised. To get up and running, I had to force the plugin list to refresh manually.

Manage Jenkins > Manage Plugins > Click the Advanced Tab

If you're like me and want git or svn scm access, you'll want to install these plugins as well, as they are not included in the list on the jenkins-php instruction page.

Once the PEAR packages and Jenkins Plugins are installed, you're now ready to start preparing your application for Continuous Integration.

The initial configuration can be quite terse, as you will need an initial ant build file, and sample configurations for PHP Code Sniffer and the other code analysis tools. Thankfully Sebastian Bergmann - author of PHPUnit and much of the Jenkins PHP suite of tools - has developed a project wizard utility to simplify these initial configuration steps.

Install the PHP Project Wizard from the PHPUnit channel:

$ sudo pear install phpunit/ppw

Once installed, you can change into your project dir and run it with a few arguments to setup your initial build environment.

$ ppw --name 'My Project' --source ./lib --tests ./tests

You can also specify arguments defining default rulesets for PHP Code Sniffer and PHP Mess Detector. Omitting these arguments sees ppw select some sane defaults for you.

Now install the Jenkins PHP Job Template:

$ cd path/to/jenkins/jobs
$ git clone git://github.com/sebastianbergmann/php-jenkins-template.git php-template
$ chown -R jenkins:nogroup php-template/
$ curl http://localhost:8080/reload # set this to be the path:port to your jenkins server

Jenkins and your system are now ready to manage a PHP Project. Please see my tutorial on Setting up a PHP Project in Jenkins for how to setup your first project in Jenkins.

It's similar with programming languages, some familiar constructs will work exactly the same as you expect. Some wont work at all, and in some cases, they'll sort of work. 

In Ruby, if you're like me and come from a C, Unix or PHP background, one of the first things to trip you up will be ARGV.

In PHP ARGV works like this:

<?php
// test.php
echo $_SERVER['argv'][0] ?> 
$ php test.php 1234 
> test.php

C and BASH work the same way.

Ruby does things a different (PERL) way:

#!/usr/bin/env ruby 
# file test.rb 
puts ARGV[0] 
$ ruby test.rb helloworld 
> helloworld 

The difference is, in PHP, C, BASH the first element of ARGV is the program's name. In Ruby, and in PERL, it is the first argument passed into the program. 

I'm trying to think which makes more sense, probably the Ruby/PERL implementation. I'm used to starting at index 1 though. So just bear in mind that even though constructs may be similar between languages, there is some devil in the detail.

Actually, it is a little bit like clicking 'start' to shutdown oin windows. To uninstall a plugin / feature you need to go to the install new software screen. On a Mac it's found by highlighting the help menu and selecting "Install New Software".

In the resulting window, there's a link which lets you view already installed software.

Clicking that link brings you to the installation details page which (assuming there are no dependency issues) lets you uninstall unwanted features.

Uninstall the unwanted software and restart Eclipse and you're good to go.

This can manifest itself in a number of ways, but the most common, I think, is what happened with me

aaron ~/Development/ruby/testapp $ rake db:create (in /Users/aaron/Development/ruby/testapp) Couldn't create database for {"reconnect"=>false, "encoding"=>"utf8", "username"=>"dbuser", "adapter"=>"mysql", "database"=>"testapp_development", "host"=>"127.0.0.1", "pool"=>5, "password"=>"testpwd"}, charset: utf8, collation: utf8_unicode_ci (if you set the charset manually, make sure you have a matching collation)

Seems the mysql driver gets confused a little bit as when you install the MySQL Rubygem as directed by Rake, you link against the bundled OSX MySQL and not the Macports one.

The solution is to install the mysql gem as follows (uninstalling it first, if necessary)

sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/opt/local/bin/mysql_config5

Substitute the mysql_config5 path with your own macports mysql_config path. Afterwards, everything should work fine.

aaron ~/Development/ruby/testapp $ rake db:create (in /Users/aaron/Development/ruby/testapp)

// Retrieving documents with find() method using a query string
$query = $idFieldName . ':' . $docId;
$hits  = $index->find($query);
foreach ($hits as $hit) {
    $title = $hit->title;
    $contents = $hit->contents;
}

This proves _painfully_ slow, you're loading the full index in an attempt to find a unique document with an ID. Even worse is if your unique ID happens to be a string such as a url or path. Since ZF 1.5, the 'best practice' direction to perform this type of task is to use the Zend_Search_Lucene::termDocs() method:

$term = new Zend_Search_Lucene_Index_Term('/somepath/somewhere', 'path');
$docIds = $index->termDocs($term);
foreach ($docIds as $id) {
    $doc = $index->getDocument($id);
    $title = $doc->title;
    $contents = $doc->contents;
}

Performance wise this proves much more efficient. However, unless you're careful at the indexing stage you may run into trouble when running termDocs() on a string value such as a URL or path as opposed to an integer ID. This is down to the field being added tokenized. This is the most common way fields are added and corresponds to:

$doc = new Zend_Search_Lucene_Document();
$doc->addField(Zend_Search_Lucene_Field::Text('title', $title));

If you want to use termDocs on an identifying field you need to add the field as type Keyword:

$doc = new Zend_Search_Lucene_Document();
$doc->addField(Zend_Search_Lucene_Field::Keyword('http://a.com/uri', 'uri'));

Keyword fields are not tokenized, and a term vector (which termDocs() requires) is stored, the distinction between the two field types is documented in Zend_Search_Lucene_Field's phpdocs:

Zend_Search_Lucene_Field::Text() constructs a String-valued Field that is tokenized and indexed, and is stored in the index, for return with hits. Useful for short text fields, like "title" or "subject". Term vector will not be stored for this field.

In contrast see:

Zend_Seach_Lucene_Field::Keyword() constructs a String-valued Field that is not tokenized, but is indexed and stored. Useful for non-text fields, e.g. date or url.

This caught me out a little bit until I dug around the source a little bit looking to see where termDocs was going wrong. Hopefully this helps save someone else some time, and hopefully Zend can update their documentation to draw other developers' attention to this quirk.

If you are looking for the long $filters for ReflectionClass::getMethods(), here they are. They took me a  long time to find. Found nothing in the docs, nor google. But of course, Reflection itself was the final solution, in the form of ReflectionExtension::export("Reflection").

// The missing long $filter values!!!
ReflectionMethod::IS_STATIC;
ReflectionMethod::IS_PUBLIC;
ReflectionMethod::IS_PROTECTED;
ReflectionMethod::IS_PRIVATE;
ReflectionMethod::IS_ABSTRACT;
ReflectionMethod::IS_FINAL;

// Use them like this
$r = new ReflectionClass("MyClass");
// Print all public methods
foreach ($r->getMethods(ReflectionMethod::IS_PUBLIC) as $m) {
    echo $m->__toString();
}

Another example, this time one of my own, is one that I found myself writing while working with the Zend Framework's Zend_Controller implementation:

/**
 * @param   String $controller_class
 * @return   ArrayObject
 */
private function getActionList($controller_class)
{
    $reflection_class = new ReflectionClass($controller_class);
    $methods = $reflection_class->getMethods(ReflectionProperty::IS_PUBLIC);
    return new ArrayObject($methods);
}

Like many platforms, in PHP it seems documentation is no replacement for digging around the source itself.

Firstly you need to backup your forum database, the simplest way is to make use of the mysqldump utility.

mysqldump -uusername -ppassword forum_db_name > forum_db_backup.sql

Copy this output somewhere safe just incase of disaster. You can always revert to what you had by reimporting from this dumpfile. By default VBulletin applies the 'latin1' charset to its schema. We want to replace all instances of this in the dumpfile with 'utf8'.

sed -i 's/latin1/utf8/g' forum_db_backup.sql

Now it's time to take your forum offline. Navigate to: AdminCP -> VBulletin Options - > Turn your VBulletin on/off and change the forum active option to off. When this is done, just for good measure, 'chmod 000' the forum root directory.

Drop the old VBulletin forum database and create a new, utf8 friendly, one.

mysql --user=username --password=password --execute="DROP DATABASE forum_db_name; CREATE DATABASE forum_db_name CHARACTER SET utf8 COLLATE utf8_general_ci;"

It's time to convert the old database dump to use the UTF-8 character set. My favourite utility to do this is iconv. This util can easily be installed using yum, apt, emerge, synaptic or whatever package manager your distro relies on

iconv -f latin1 -t utf-8 forum_db_backup.sql > forum_db_backup-utf8.sql

Pump the fixed SQL back into your database.

mysql -uusername -ppassword forum_db_name < forum_db_backup-utf8.sql iconv -f iso8859-1 -t utf-8 vbulletin-language.xml > vbulletin-language-utf8.xml

Then import the language in the usual way AdminCP -> Languages & phrases -> Download/upload languages.

It's good to think of your CV as like a tool, no different from a programming language or an IDE. Like any tool it has to be rigorously tried, tested and possible refactored if it is to be successful and help you achieve your goals. In the case of a CV its purpose is two-fold, a) to get yourself past a recruiter and onto the actual client and b) to impress the client sufficiently that they want to interview you. So in the spirit of the new year and everyone having 'ten things' lists, here's mine for what to avoid in a modern CV for an IT role:

1. Don't send an old CV. If you haven't updated your CV in months, let alone years, it isn't going to sell you as effectively as it can. As a developer or even a recent graduate you're learning all the time. Detail and demonstrate as much of that learning (that is pertinent to the job at hand) as possible. As you get older, more experienced, your perception of what is and isn't important improves. While recently reviewing my own CV I found I had added a significant amount of detail, while actually reducing the overall bulk of the document.
2. Please, do not include an interests section. This point is somewhat subjective I grant. But of the CVs we've been forwarded all but one had an interests section. Not one of these contained anything relevant to the job being applied for, and in some cases actually contributed to us ruling the candidate out. When hiring, a client is seeking a professional. Telling them you spend a lot of time playing 'World of Warcraft' in your spare time isn't going to get them jumping out of their skin to interview you - particularly if you don't qualify how that interest is relevant to you performing the duties required.
3. Avoid sending a generic CV. Applying for jobs is tedious, time consuming and not immediately rewarding. But sending a generic CV detailing every subject you did at GCSE and A levels doesn't tell the client why they should hire you, and you specifically, for the job at hand. Oh and most clients seeking developers don't care if you worked at Subway or have a Tesco certification in fish mongering, it's simply not relevant.
4. Web 2.0 extravagance might be 'in' right now, but try to keep your CV conservative, well spaced and using a minimum of fonts. Giant mastheads, fancy bullets, a giant mess of fonts, it isn't impressing anyone, seriously, and most recruiters gut your CV into a standardised format anyway. Do be mindful of how it is to read. Clean and clear is the key, but save the extra special effort for the content.
5. Don't go on for longer than two pages. Trying to cheat this rule by making the font so small that a microfiche reader is required is only going to annoy the recruiter. That means your CV gets forwarded to the 'round file' not the client's inbox.
6. Don't list every course you did at University and expect the client to care. Of our graduate applicants not one has detailed what the courses actually contained. Database Systems 1. Cool - and you did what exactly in that course that is relevant to the job? It's often difficult for graduates to pad out their CV in the same way as someone with a few years' experience. Treat your degree like a job, work out what specific skills you got from your courses and detail the skill qualified with a practical example of how you applied it in your course work. Start doing that for all of your courses and very quickly you'll bang into the two page rule. Keep going, but when you're finished pare down the text into bullet points that are directly relevant to the job spec.
7. Spelling mistakes! Simply do not forget to proof read your CV. This really ought to go without saying. Get your sister, mother, girlfriend, house mate, anyone, to read your CV before sending it. Read it backwards, turn the zoom up to 400% and read it one word at a time. Anything to ensure that what you send out is as professional as you are. A sloppy CV leads to a sloppy impression. 'The Halo Effect', while not always fair, is something you have to deal within the real world. If a client's first impression of you is a bad one, all subsequent impressions will be shaded by it.
8. Don't include a picture of yourself. Like ever. It's simply not necessary.
9. Don't ramble, don't generalise, don't make statements without qualification. Good use of English is one of the best indicators of a good developer. Good code is succinct, clear and well qualified (unit tested). Like PHP, Java, indeed any programming language, English has a grammar, a vocabulary and patterns of use. Make best use of these attributes in your CV. Constantly refactor your CV to weed out redundancies, group like concepts, simplify and support the concepts. Like good code, a good CV goes through many iterations until it's 'good enough' for the real world. Mercilessly refactor your CV as you would your program code.
10. Don't accept everything you read on the net about CVs as gospel. This caveat applies to everything in life and not least this blog itself. If you listen to every top ten list of what not to include in your CV you'll quickly find out there's absolutely nothing you should put in your CV. Critically consider what you read suggested about what a good CV looks like and make your own mind up based upon the supporting arguments made and your own CV's feedback. For example if you disagree about point two and decide to include an 'interests section' ask recruiters when they call you what they think about it, did it provide value or was it noise? If you're getting interviews ask the recruiters what in your CV is standing out. If you're not, then ask what, if any, feedback there is from the client - and react to it.

I really do think of a CV like program code or a developer's utility. As a good developer you always need to evaluate your tools to ensure they are providing the maximum value. And like program code there is always an element of entropy to catch and counter over time. A CV, like program code, can sometimes be brittle and break with additions. Using and maintaining one is a matter of constantly reviewing, refactoring and responding to user feedback. For some good resources on CV writing I really recommend the following links: http://www.harveynash.com/uk/hnit/jobs/articles_cv_basics.asp http://www.lifeclever.com/give-your-resume-a-face-lift/ And for graduates especially: http://www.kent.ac.uk/careers/cv.htm

To guide me through the Ruby syntactic jungle is O'Reilly's 'Learning Ruby' book. I checked out the ubiquitous Pragmatic Programmer's 'Programming Ruby', but the third edition seems more suited to stopping doors than teaching Ruby.

Reading through the first few chapters of 'Learning Ruby' has left me a little nostalgic of the early 90s when I was going through my first programming book 'Learning Perl'. Maybe it's the title similarities but mostly it's that Ruby's syntax feels as slippery to me as Perl's did.

I don't mean that in an endearing way.

While I'm in love with blocks (or closures whatever you prefer) and Procs and Mixins (Traits), a simple thing scares me. Constants.

Constant, the word is pretty clear in its meaning. It is a value which remains stable over time. Wikipedia says: 'a constant is a special kind of variable whose value cannot typically be altered by the program during its execution'. Collins dictionary says: ‘something that does not or cannot change or vary’.

Ruby though, says no. Constants are just global variables declared capitalised. Oh sorry that should be capitalized. Ruby doesn’t speak English, only American. A constant in Ruby is about as constant as your discipline. If you REALLY want to change its value, Ruby wont stand in your way, or even make it that hard to do.

That really, really concerns me. Part of the appeal of Ruby, I guess, is that it lets you do things you want, your way. Maybe it’s ‘Fear’ but I really don’t want to have to worry about my constants changing. Ruby makes a big deal of its ‘duck typing’ You know, if it quacks like a duck, it is a duck. Unfortunately in Ruby a constant quacks like a duck but it bites like a hippo.

This is, of course, not absolute, there are camps of reformed Perl sys-admin monkeys that haven’t been seduced by Python. And of course a rare breed of ex systems programmers versed in C and C++.

It is actually this last group of coders that tend to get on with PHP the best.

This makes sense, because PHP was originally designed to be a template engine to produce HTML with a core C backend doing the grunt work. This seems a ridiculous idea now, as CPU cycles are much cheaper than man hours. But in the late 90s native code ran circles around dynamic, interpreted alternatives.

These days C is a little bit like ASM was in the 90s. Being proficient in it is handy, but not essential. I would argue though, for PHP coders it is an important environment in which to be familiar.

Many of PHP's eccentricities can be traced to it’s C roots. The strange and inconsistent function parameter orders, right down to the sometimes peculiar way it handles memory and references.

The file and string functions marry up almost 1:1 with standard C. Experience of and understanding of how these low level functions work will immeasurably improve your application of their PHP counterparts.

Networking is another area where PHP and C are very similar. Indeed you can write a simple tcpclient in PHP, get rid of all the $ variable identifiers and have a recognisable C fragment.

The biggest benefit though of learning a bit of C or C++, is to get an “appreciation” of memory management. C is unique among the programming environments of today in that it does not manage memory for you. Arguably this is what makes C so painful to develop in. But similarly, a sometimes useful characteristic that makes the environment still relevant today.

By way of example, strings in C are represented by an array of chars, terminated by a NUL ‘\0'.

In C, you would declare a string like  this: char str[20];

This declares a string of 20 characters, and str itself points to an address in memory where 20 bytes have been reserved for itself. Now if we try to write 21 characters to this string, C wont autoexpand the array to fit, or do anything so helpful as to warn you that you’ve gone out of the bounds of your allocated storage. No, C will do what you tell it, and in this case it means overwriting someone else's data.

This is quite a powerful characteristic and as such requires a programmer to be responsible.

Generally if you have a 20 char string defined but want to store a 25 char string, you need to reallocate memory. You could declare a new array or do a concatenation operation. Either way, this costs CPU time.

So when in PHP you are spamming '.' concat operators everywhere, you can appreciate that PHP is doing a lot of memory re-allocation under the hood to provide that syntactic sugar.

As mentioned above, PHP was originally intended to be a purely templating language for C web applications. Which is where PHP modules / extensions come in. These originally were where your business logic was to go. PHP extensions are compiled to native code, and as such run FAST. You can then access these functions in PHP just like the other core functions and classes. Yahoo, Facebook, use PHP in this way.

So can you. A great way to dabble in a bit of C is to write your own PHP extension. The benefit of getting your hands dirty in this way is the nice speed boost for the functions in your extension. But the more important benefit is how it can greatly improve your understanding of what PHP does for you under the hood. This lets you make informed design choices when writing your regular PHP code.

As a starting point, I can thoroughly recommend starting out with the Zend engine hacker's guide on the main PHP site. Becoming familar with the structure of the internals is the first step to appreciating all the of the magic PHP does for you under the hood.

But how does this affect you as a developer, and what is the reasoning behind each different approach. This paper goes a long way to explaining this, rather fundamental, aspect of computer science.

This can be inconvenient at times when you already have the data sitting in a variable. A way of getting around the need to worry about physically creating a file is to use the memory stream type.

PHP supports a number of input / output streams ranging from the usual stdin, stderr, stdout to memory, temp and filter.

See http://php.net/manual/en/wrappers.php.php for more information on these.

But looking at the memory type, it's very easy to use. Simply $fh = fopen('php://memory', 'wb+'); and you can use the usual file functions you would typically associate with an ondisk file.

You can fread, fwrite, file_get_contents on the memory stream or push it out over the network using the tcp streams PHP offers. PHP streams are a powerful and often underutilised aspect of the language.

SELECT DATE_FORMAT(yourdate, '%Y-%m') AS grouping_date COUNT(id) FROM yourtable GROUP BY grouping_date;

Change the date format to whatever date quantum you want to report on, i.e. just have '%Y' for a yearly grouping, or have '%y-%m-%d' for a daily one.

The quickest way, is actually very simple, and that is to simply use curl to resubmit the callback. This is assuming you have found, and if necessary corrected, the issue that stopped the callback succeeding in the first  place.

When a callback fails, either through a time out or some internal (500) error, you will receive an email from Worldpay notifying you. This mail includes two attachments one the request data Worldpay sent to your callback URL including the encoded post data as well as the response from your server.

Now assuming you work around your security arrangements (e.g your callback URL should listen only to certain addresses and expect auth credentials), you can use curl to resubmit your post data:

curl -d ‘testMode=0&authCost=716.86&currency=GBP&address=15+Somewhere+Street&countryString=United+Kingdom&callbackPW=xyz&installation=2555555&fax=&countryMatch=Y&transId=1000000000&AVS=2222&amountString=%26%23163%3B716.86&postcode=XX11+1XX&msgType=authResult&name=Mr+Test+Tester&tel=0208+111111&transStatus=Y&desc=Test+Transaction+1234&cardType=Visa+Delta&lang=en&transTime=1277741069650&authAmountString=%26%23163%3B716.86&authAmount=716.86&ipAddress=80.80.80.80&cost=716.86&charenc=UTF-8&instId=2555555&amount=716.86&compName=Testing Tester&_SP.charEnc=UTF-8&country=GB&rawAuthMessage=cardbe.msg.authorised&authCurrency=GBP&email=testy@tester.com&cartId=12345678&rawAuthCode=A&authMode=A’ \
'https://mysite.com/callback'

Now your program can handle the completion of the order as normal.

You can make this more sophisticated, by having a mail reader retrieve any missed payment notifications and parse them. With each failure notification you can add a retry to a job queue, or other batch processing stack. It would be worth adding some sort of retry threshold though, as well as sending notifications to developers to ensure there is no serious malfunction occurring.

For the impatient, the cli programme that weaves this magic is called 'textutil'.

But  anyway, using svn:externals to include external libraries is a convenient way of moving third party libraries out of your own repository.

For example, many PHP applications today, have the Zend Framework as a dependency. To include the library within your application without actually having to download the tarball and use something like phing to set it up in place, you can simply do a

svn propedit svn:externals

Then in your editor, add the following line (each externals definition goes on a new line)

Zend http://framework.zend.com/svn/framework/standard/tags/release-1.10.5/

Set this property on your standard application library directory, or somewhere in your include path.

When you run svn update, svn will pull down the Zend library into your defined location.

The Zend Framework, out of the box, comes with some dojo dependencies that are optionally bundled with the framework. You can use the same approach to pull in the dojo deps by setting svn:externals definitions for your javascript include dir and pointing them at http://svn.dojotoolkit.org/src/tags/release-1.4.3/ (or trunk if you're brave).

Again, there are some good arguments for NOT doing this, but for me, the convenience of this approach outweighs those (which I'll cover in more detail later).

The first one is the one you're most likely to encounter after upgrading

 Fatal error: Call to a member function toHtml() on a non-object in <somewhere>/app/code/core/Mage/Core/Model/Layout.php on line 52

The solution is explained here:

http://rackspeed.de/forum/magento-faq-installation-and-updates/fatal-error-call-to-a-member-function-tohtml-on-a-non-object-layout-php-on-line-529-a-547

Basically edit your theme's layout/page.xml and change <block type="core/profiler" output="toHtml"/> to <block type="core/profiler" output="toHtml" name="core_profiler"/>

The next two are frankly amazing in that they ever made it into the release.

See here:

http://www.magentocommerce.com/boards/viewthread/195761/

To quote the finder of these issues 'Yes, preparing a shipment prepares an invoice instead..'

In short, when overriding a helper, the context element IS case sensitive. With blocks, it is NOT.

<![CDATA[
{{depend prefix}}{{var prefix}} {{/depend}}{{var firstname}} {{depend middlename}}{{var middlename}} {{/depend}}{{var lastname}}{{depend suffix}} {{var suffix}}{{/depend}}|
{{depend company}}{{var company}}|{{/depend}}
{{var street1}}|
{{depend street2}}{{var street2}}|{{/depend}}
{{depend street3}}{{var street3}}|{{/depend}}
{{depend street4}}{{var street4}}|{{/depend}}
{{depend city}}{{var city}},  {{/depend}} {{var region}} {{depend postcode}}{{var postcode}} {{/depend}}|
{{var country}}|
{{depend telephone}}T: {{var telephone}}{{/depend}}|
{{depend fax}}<br />F: {{var fax}}{{/depend}}|
]]>

The format may be slightly terse but hopefully you can see how it can be applied to create your own desired format.

With magento models, you may be tempted to use camelcasing for long class names, for example MyPackage_MyModule_Model_ALongNameForAModel. This will not work as you might expect on different (namely case sensitive) environments.

Your Mage::getModel(mymodule/a_long_name_for_a_model) call will translate to the class loader trying to find A/Long/Name/For/A/Model.php.

Conversely trying to address your model Mage::getModel(mymodule/alongnameforamodel) will see the classloader trying to load Alongnameforamodel.php, and not ALongNameForAModel.php.

On Windows this is fine, on case-sensitive e.g., HFS (Mac) or Unix file-system, this will not work.

UPDATE cms_page SET custom_theme = 'custom/default' where custom_theme='default/custom';

As one of my clients' sites has grown, some batch updates were taking up to 30 minutes to run. This is too long.

If the changes you are making are just to update simple attributes, for example we have a sales ranking attribute, you can use the following code to update the value without incurring the massive overhead of a full product save.

$product->setNumSales(1234); $product->getResource()->saveAttribute($product, 'num_sales');

The saveAttribute method takes two parameters, the first is the model containing the attribute value, the second the attribute code. To find out the attribute code, look it up in either the db (eav_attribute) or in the admin backend under catalog->attribute.

Using the getResource()-->saveAttribute() call, takes 1/5s, doing a full save(), takes 2-3 seconds. When iterating over a large product base, that is HUGE.

Update 4 Mar 2014 - Please take a look at DannyD's comment below for a more robust approach to mass attribute updates.

For example, say you want to select all records from a table that has a date set more than 1 month old (compared to the current time).

SELECT * FROM table WHERE DATE_SUB(CURDATE(),INTERVAL 30 DAY) > date_column

This assumes your date column is of type date, or datetime, although using FROM_UNIXTIME will allow you to work with timestamps as well.

The date_sub call, if ran on 2010-05-20, will return 2010-04-20. So if your date_column field has a date greater than this value, it is therefore more than 30 days old.

This is particularly handy when generating reports.

I just saw an insane suggestion on a Magento forum, advising users to change their admin url by hacking app/code/core/Mage/Adminhtml/etc/config.xml.

This is dangerous and pointless. Dangerous, because when you attempt to upgrade magento, you will at best lose your custom configuration, and at worst, cause an upgrade to fail. Pointless, because you can achieve the same effect by making changes to app/etc/local.xml.

This file overrides any configuration set elsewhere and by default, actually already has a definition for a custom admin url. When you install Magento, the installer sets up this file up by subbing in the details you provide the installer into local.xml.template.

So to change your admin url, after running the installer, simply fire up an editor and open app/etc/local.xml and look for this:

<admin>
    <routers>
        <adminhtml>
            <args>
                <frontName><![CDATA[admin]]></frontName>
            </args>
        </adminhtml>
    </routers>
</admin>

Change the admin (inside the CDATA declaration) to foobar, or gobbles, or topsecret, whatever. Much easier, much safer than hacking away at core magento configuration files.

To add a text template to a view in magento, you need to consider three aspects

1. The template block itself
2. The block, or layout, the template will sit in
3. The layout .xml configuration

The first element is simplest, create some html, stuff it into a .phtml file and copy it to a directory within your theme, which resides (relative to the store root dir) in 'app/design/frontend/<package>/<theme>/templates'. I'm going to assume the package/theme is default/default from here forward.

You will now need to configure one or more layout xml templates (app/design/frontend/default/default/layout') to refer to your new template block. For most general purpose, globally available blocks, this will be 'page.xml'.

For example, to add a productfinder template to the three column layout, you need to edit page.xml and within the

<block type="page/html" name="root" output="toHtml" template="page/3columns.phtml">
...
</block>

block add the following code near the closing </block>:

<block type="core/template" name="my_product_finder" as="my_product_finder" template="templatedir/productfinder.phtml"/>

Type="core/template' refers to a Magento class Mage_Core_Block_Template. This type can be different if you want a specific type of block. But core/template is the simplest way to include some text on a page. The 'name' and 'as' attributes allow you to reference the block in your enclosing templates, e.g. 3column.phtml, or other layout .xml files. The template attribute is the relative path to the text template you want to include and is relative to the theme template root i.e. app/design/frontend/<package>/<theme>/templates.

Once you have created a layout definition for your block, including the template on a page is easy. In 3column.phtml, for example, simply put <?php $this->getChildHtml('my_product_finder') ?> and the text will be included. Remember to refresh, or clear Magento's cache first though.

One more element to consider is exclusions. If you put the above code in 3column.phtml, that text will appear on every page that uses that layout template. You may not, for example, want to include it on customer dashboard pages. In this case  you will need to define a remove statement within the layout.xml for the group of pages you want to change. In this example, we need to edit customer.xml

In customer.xml (and indeed page.xml) you will see a number of elements similar to <default></default> (which applies to ALL pages), <customer_account></customer_account> (which refers to all pages with the url customer/account/), and <customer_account_edit></customer_account_edit> (which refers to all pages with the url customer/account/edit/). Including your remove declaration in the right element here allows fine grained control over which pages your blocks appear.

In the example here, we want to remove the product finder from ALL customer account pages. Therefore within the <customer_account> element we add the code

<customer_account>
...

<reference name="root">
<remove name="my_product_finder"></remove>
</reference>

</customer_account>

The reference name=root element ensures we are altering the root block, which is the one in which we defined our template block in page.xml. The enclosed remove call, uses the name of the block we created "mns_product_finder" to identify the block we want to remove for these pages.

That's it! All pages that exist under <customer_account> (so all of them) have this block removed.

To see if you are affected run memcached in -vvv mode and look out for this:

CLIENT_ERROR bad command line format.  Usage: delete <key> [noreply]

This causes magento to throw an exception with the message ""can't get stat from localhost:11211".

The problem is down to a change in behaviour of memcache in version 1.4 where they removed the second parameter passed to the delete function. See here for more info http://code.google.com/p/memcached/wiki/ReleaseNotes144. 1.4.4 thankfully added some backwards compatibility.

Magento 1.4 uses Zend Framework 1.9.6 and this, sadly, does not support these changes. Version 1.10.3+ does.

http://framework.zend.com/issues/browse/ZF-9376

I use iTerm as my terminal of choice, and it natively supports utf-8. However by default, your terminal environment does not. I have my environment configured to use the de_DE (German) locale. However some characters, most typically ü and ä do not display correctly.

In order to get correct text in your utf-8 terminal you need to configure your environment to use a utf-8 enabled locale. If you're using en_GB, or de_DE (and you can get a list of available locales by calling 'locale -a') you just need to edit /etc/profile or ~/.profile or ~/.bash_profile, and put the line export LC_ALL='de_DE.UTF-8' or LC_ALL='en_GB.UTF-8'

This will ensure your terminal AND environment are speaking the same language.

This happens when you try to render non utf-8 text as utf-8. Utf-8, shares the same 7-bit ASCII range as common windows encodings cp-1252 and iso-8859-1. Lower ASCII is the basic alphabet e.g. a-z, A-Z, hyphens, commas, etc. However extended characters, like accents, symbols and umlauts change. So a cp-1252 trademark symbol has a different code in utf-8. Try to render a cp-1252 copyright on utf-8 you will just see a question mark in the browser, as that code does not have a corresponding entity in utf-8.

To fix this you need to normalise your content. If it's going into mysql, ensure your tables are set to your normalised format. I recommend using utf-8 as this normal form.

To normalise your text, use iconv, which is available as a php extension.

$utf8Text = iconv('iso8859-1', 'utf-8', $isoText);

When outputing to the browser run this through htmlentities or htmlspecislchars. Both these functions expect iso-8859-1 input. To correctly prepare your utf-8 text for output you need to supply a third parameter to these functions supplying your text's encoding, in this case utf-8.

htmlentities($text, null, 'utf-8');

Failing to supply this parameter means your text will be broken and your extended characters displayed as questiomarks.

PHP provides all the tools you need to properly handle many different text encodings. As a developer you just need to normalise your input, and properly encode your output.

From the man and help pages it would seem doing sed -i 's/hello/goodbye/g' helloworld.txt would be the way to achieve it.

It doesn't, and you'll either get a script processing error or something like: sed: -i: No such file or directory.

The trick, is to use it like this: sed -i '' 's/hello/goodbye/g' helloworld.txt 

Still, it's very easy to get the baseurl directly from magento, below I've summarised the most common ones you'll want.

Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_WEB);
http://myshop.mydomain.com/

Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_LINK);
http://myshop.mydomain.com/index.php/

Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_JS); http://myshop.mydomain.com/js/

Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_MEDIA);
http://myshop.mydomain.com/media/

Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_SKIN);
http://myshop.mydomain.com/skin/

So, say you want to create a link to your store's checkout page you would use the following code:

<a href="<?php Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_WEB);?>checkout/onepage/">Checkout</a>

It's well worth taking a look at the getBaseUrl function in the Mage class(app/Mage.php) and Mage_Core_Model_Store (app/code/core/Mage/Core/Model/Store.php)

I had an array of files that I wanted to do some operations on and the following construct allowed me to easily iterate through that list.

FILES=( a/path/to/a/file1 a/path/to/another/file2 and/so/on/and/so/on )

for ELEMENT in $(seq 0 $((${#FILES[@]} - 1))); do
  echo ${FILES[$ELEMENT]}
done 

The $(seq 0 $(($#FILES[@]} - 1))); returns the number of elements in FILE, the seq command produces a sequence of numbers from x to y. If you call seq 0 4, you will get a line with 0, 1, 2, 3, 4 on it.

So while the syntax is a little smelly, the terse power of it, is quite handy.

Often, you will want to change some details, baseurls, use test payment or shipping account details.

Generally this is pretty easy to script up in bash, python or even plain php. However magento encrypts some of the data it stores in core_config_data.

The following is the approach I use to update data in core_config_data  in an encrypted format magento will accept:

<?php
require_once 'app/Mage.php';
umask(0);
$app = Mage::app('default');
$data = $_SERVER['argv'][1];
$obj = Mage::getModel('core/encryption');
$helper = Mage::helper('core');
$obj->setHelper($helper);
echo $obj->encrypt($data);

I use it as part of my git workflow (see my magento githooks repo) and I find it really does help keep code consistent.

When you work on projects with more than a couple of developers, niggling differences in style can lead to unintentional errors creeping into your codebase (think omitting braces in if statements, for example).

Anyway, one of the neat features in the master branch is the ability to specify a config file to control how php-cs-fixer behaves rather than having to define everything on the commandline (or depend on .php_cs being in the path). Unfortunately this feature is not yet in a pre-baked release of php-cs-fixer. So to use it you have to use the source version. I quite like the convenience of the phar version but it's unclear how to build one directly from the sources.

After a bit of digging around in the code and issues list, I found they are using the php-box project to build release phars. It's actually very simple, but to save others having to figure it all out, just follow these steps.

$ git clone https://github.com/fabpot/PHP-CS-Fixer
$ cd PHP-CS-Fixer
$ composer.phar require --dev 'kherge/box=~2.4'
$ vendor/kherge/box/bin/box build
> Building...

Do a quick ls and you'll notice you have a minty fresh php-cs-fixer.phar file.

Done.

$ usermod -a -G thegroup theuser

And job done. But OSX uses Open Directory rather than traditional flatfiles like /etc/passwd and /etc/group to store information about users and domains. So the typical unix commands we are used to, don't work.

The dscl (directory service command line) utility lets you manipulate Open Directory values and in our case, add a user to an additional group, handy if for example, you want to add your user to the wheel group to make use of password free sudo.

$ dscl localhost --append /Local/Default/Groups/<groupnamehere> <usernamehere>

So, later on, you'll probably want to pull changes down from the remote and you'll end up seeing something similar to this

➜  store git:(zendesk) git pull --rebase                                                          [48/53]There is no tracking information for the current branch.
Please specify which branch you want to rebase against.
See git-pull(1) for details

git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

git branch --set-upstream-to=origin/<branch> zendesk

Now it's not that hard to type out the suggested command above to set the upstream branch, but I got sick of having to do it so often, and I have given up trying to remember -u, so I created a git alias to automate things and save some keystrokes.

In your ~/.gitconfig under the alias section, add this

    sup = !git branch --set-upstream-to=origin/`git symbolic-ref --short HEAD`

You can use the alias by issuing the following command in your terminal

$ git sup

This will look at the current branch and set its upstream to origin/branchname

If you tend to use another remote name other than origin, change the alias accordingly.

I have a few other useful aliases which you can checkout (hah, sorry :)) in my full gitconfig.

For some workarounds checkout http://mywiki.wooledge.org/BashFAQ/024

Usually these types of commits can be avoided by keeping feature branches up to date with git --rebase. When two branches have a direct common history, merges can be applied using the fast-forward strategy avoiding the need for a stitch-things-together merge commit.

Because the commit pointed to by the branch you merged in was directly upstream of the commit you’re on, Git moves the pointer forward. To phrase that another way, when you try to merge one commit with a commit that can be reached by following the first commit’s history.

http://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging

To ensure you've kept your branches synced up with rebase and to avoid accidentally creating a merge commit, you can set git merge to only perform fast forward merges.

$ git config --global merge.ff only

This way, you'll get a gentle reminder to rebase. If that's not feasible then you can force through the merge with

$ git merge --no-ff

• grumpy-phpunit.com
• Price: $29.00

The Grumpy Programmer (actual name Chris Hartjes) amusingly blogs and tweets all things PHP and particularly PHPUnit. When I saw he was publishing this book, I was curious to see how his strident style would stand up to the longer form. Pleasantly, it turns out.

Chris maintains his gruff voice while whirling through the ins and outs of using PHPUnit. I've been using PHPUnit for a long time now and I find that when I am really familiar with a tool, I tend to re/(over)use patterns that have served me well in the past. So while I found I was familiar with much of the material in the book, there are more than a few tidbits here that I picked up. I feel even the most grizzled PHPUnit veteran's testing regime will benefit from a read through.

The book seems aimed at the less experienced which I did find a little surprising given the title. When I think cookbook I tend to think of the weighty O'Reilly tomes. This book though is more like a lengthy tutorial than a cookbook in the O'Reilly style. As a tutorial goes though it excels. It is detailed without being turgid and covers all the major aspects of using PHPUnit that I would expect it to and then some. I found the chapter on Test Doubles (that is mocks, stubs and fakes) to be particularly excellent. The vocabulary surrounding these terms tends to get mixed up and consequently programmers often treat them as the same thing. That leads in my experience, at best, to confusion. And at worst, to poor tests that are difficult to maintain.

As a quick aside, the book is published by LeanPub, who ensure authors receive 90% of the proceeds from their work. I think this is a wonderful initiative. Writing, especially for a programmer is tremendously hard and I like the idea of those that attempt it, and do a good job, get appropriately rewarded for doing so.

So, back to the book. You find peppered throughout the introduction to PHPUnit subtle wisdoms that are hard to argue with. A simple and you would think obvious example, is that of always providing the final argument to assert statements with a description message. This message is displayed when the test fails, helping you quickly identify where the problem lies. Another: writing strictly encapsulated code that eschews static methods and class variables, is (well unsurprisingly) easier to test that code that is constantly mutating global state.

The book is quite short, coming in (at least in my pdf version) at 85 pages. I feel like there is sufficient scope for more content here. Especially for a 'cookbook'. I would have loved to have seen more on using Data Builders for example. The chapter on data providers is great, but I find you often need more fine grained control over your fixtures. Factories and data builders are a couple concepts that once learned, significantly reduce the friction of TDD.

I perhaps would also liked to have seen more in introduction to TDD itself, motivations for it, and perhaps a brief comparison between the two principle TDD xUnit styles. Specifically the Statist TDD and Mockist/London School TDD styles. The former being a test style mainly interested in setting up some state, running a behaviour and checking the end state matches what you expected. The Mockist approach is less interested in observing State and instead is more interested in the messages passed between objects (method calls between collaborators).

Overall I enjoyed the book, and it fills a much needed role in guiding budding PHP TDD practitioners in the use of the most mature tool we have available in PHP. I picked up a few neat new tricks and I suspect many PHP programmers will do the same.

You can buy it now at grumpy-phpunit.com

The problem: you want to select the smallest value from a set of values.

Let's say you have a table of products that are in a logical group and you want to select the lowest priced product from that group, however some products actually have a 0.00 price (for whatever reason). You don't want to actually show 0.00 as the lowest price for this group of products, you want to show the lowest price that happens to be greater than zero.

MySQL has a neat way to do this. Simply go:

SELECT tableref.group_id, MIN(NULLIF(tableref.column, 0)) as min_price FROM tableref GROUP BY tableref.group_id;

The magic is in the NULLIF function, which will return null if tableref.column is equal to 0. Returning null removes that value from inclusion by MIN, having the effect of forcing the column value to be greater than zero.

So I'm happy with a less flexible, but attractive, functional and predictable desktop and I think many others feel the same way. It's no real surprise to me then, that OSX has mostly killed off the idea of Linux on the Desktop.

But somewhere that OSX falls severely behind, is its use of a BSD inspired Unix implementation. If you're born and raised on a diet of GNU (file|core)utils, of apt, yum, and portage, heck even sysvinit, OSX's realisation of Unix leaves a lot to be desired.

With considerable effort and some patience though, OSX can be brought to heel. With Iterm2 and Macports you can have a functional GNUlike Unix experience.

I'll go over the minutiae of my Macports setup another time, but generally speaking I replace all the default OSX tools with GNU equivalents and favour /opt/local/bin over everything else. It means I can have one set of configs which work mostly unchanged across Linux and OSX instances.

Macports is pretty good and the folks that contribute to it do a great job. But it does lack the polish that you take for granted with the Linux package managers. Another point to keep in mind is Macports, like Portage and BSD Ports, is a source-code based 'package' manager. When you install something, it is compiled right there and then on your system. When things go wrong, unless you're a competent C programmer (and even then) you're going to have a bad time.

One last thing to remember too, is OSX defaults to a case insensitive (but thankfully case-preserving) HFS filesystem. By default, PHP and php appear as the same thing to HFS.

So the point of this blog is to go over getting PHP running natively with Macports and how we can run an instance of Magento and the Magento Test Automation Framework (TAF).

MySQL

MySQL is probably the easiet part of the whole thing to setup. So let's start there. For reference, the database files are stored under /opt/local/var/db/mysql55.

In Macports MySQL carrys a namespace of sorts by the way of a version suffix (as does PHP). This lets multiple versions of a package be installed side-by-side. The drawback is rather than having a mysql command, you have a mysql55 command. That's annoying. So we will install mysql_select which lets us select a version to activate and give us proper file names.

$ sudo port install mysql55-server mysql55 mysql_select
$ sudo port select mysql mysql55
$ sudo port load mysql55-server

We will want a database for our magento application.

$ mysqladmin -uroot -p create magento 

PHP / PHP-FPM

Now we want to install PHP, PHP-FPM and the extensions Magento and TAF require.

$ sudo port install php54 php54-fpm php54-curl php54-APC php54-gd php54-pcntl php54-gd php54-mcrypt php54-iconv php54-soap php54-yaml php54-xdebug php54-openssl php54-mysql php54-pear php_select pear-PEAR

$ cd /opt/local/etc/php54
$ cp php-fpm.conf.default php-fpm.conf
$ cp php.ini-development php.ini

$ sudo vim php.ini
# set date.timezone and cgi.fix_pathinfo = 0

$ sudo vim php-fpm.conf
# make any changes for min / max num servers, error logging etc

The MySQL extension needs a little bit of prodding to look in the correct location for mysql.sock

echo 'pdo_mysql.default_socket=/opt/local/var/run/mysql55/mysqld.sock' | sudo tee --append /opt/local/var/db/mysql.ini

Once PHP-FPM is installed and configured you can use Macports to tell launchd to start it automatically.

$ sudo port load php54-fpm

PHP-Select

As with MySQL, Macports lets you install multiple versions of PHP side by side. This can be handy if you want to run PHP 5.3 and PHP 5.4 at the same time. I just install a single version, but Macports effectively namespaces everything. So rather than '/opt/local/bin/php' you have '/opt/local/bin/php54'. PHP Select, which we installed earlier fixes this by effectively 'activating' one version and creating the usual executable names we're accustomed to.

$ sudo port select php php54 

PEAR

PEAR is the single biggest pain in the whole process. And with some research it turns out its because Macports PEAR isn't even meant be used by end users (WAT?!).

There is no MacPorts port that installs the pear package manager application with the intent that it be used by the end user outside a MacPorts port install. If you want to use pear manually on your own then you should install it using gopear, composer or some other method. http://trac.macports.org/ticket/37683

So this goes a long way to explaining why Macports doesn't set PEAR up with sane defaults, or even put the pear command in the default path. But we can sort this all out easily enough ourselves.

$ sudo pear config-set php_bin /opt/local/bin/php
$ sudo pear config-set php_dir /opt/local/lib/php/pear
$ sudo pear config-set ext_dir /opt/local/lib/php54/extensions/no-debug-non-zts-20100525
$ sudo pear config-set bin_dir /opt/local/bin
$ sudo pear config-set cfg_dir /opt/local/lib/php/pear/cfg
$ sudo pear config-set doc_dir /opt/local/lib/php/pear/docs
$ sudo pear config-set www_dir /opt/local/lib/php/pear/www
$ sudo pear config-set test_dir /opt/local/lib/php/pear/tests
$ sudo pear config-set data_dir /opt/local/lib/php/pear/data
$ echo 'PATH=$PATH:/opt/local/lib/php/pear/bin' >> ~/.bashrc # or zshrc if you use zsh

Another issue you'll possibly have with PEAR, is it will default to the system PHP executable (/usr/bin/php) rather than your active Macports one. The pear command does test for an environment variable so we can set up an alias to pass this variable to pear on invocation.

Add an alias to your bashrc/zshrc in the form:

alias pear='PHP_PEAR_PHP_BIN=php pear'

Reload your bashrc/zshrc.

$ source .bashrc (or source .zshrc)

Now the alias is active we can check that it's working

$ /opt/local/lib/php/pear/bin/pear version
PEAR Version: 1.9.4
PHP Version: 5.3.15
Zend Engine Version: 2.3.0
Running on: Darwin avalanche 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

$ pear version
PEAR Version: 1.9.4
PHP Version: 5.4.12
Zend Engine Version: 2.4.0
Running on: Darwin avalanche 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

Now to make installing PEAR packages easier I turn the channel autodiscovery option on, which means you don't have to manually add channels for package dependencies (which there are a lot when installing phing or phpunit…)

$ sudo pear config-set auto_discover 1

Now add phing and phpunit and install them with all their optional dependencies and some extra packages for the Magento TAF.

$ sudo pear channel-discover pear.phing.info
$ sudo pear channel-discover pear.phpunit.de
$ sudo pear channel-discover pear.symfony-project.com
$ sudo pear install --alldeps phing/phing 
$ sudo pear install --alldeps phpunit/phpunit
$ sudo pear install phpunit/PHP_Invoker
$ sudo pear install phpunit/PHPUnit_Selenium
$ sudo pear install -f symfony/YAML

PECL/Extensions

Macports by default creates .ini files to load extensions in /opt/local/var/db/php54. If you manually build any extensions, add the appropriate ini file here, for example:

$ echo 'extension=yaml.so' | sudo tee /opt/local/var/db/php54/yaml.ini

Nginx

Apache/Nginx. It doesn't really matter. Both are great, but in production I use Nginx so I use it in development too. I install it with just the ssl extension enabled, to see the full range of available options, use:

$ sudo port variants nginx 

To install:

$ sudo port install nginx +ssl
$ cd /opt/local/etc/nginx
$ sudo cp fastcgi.conf.default fastcgi.conf
$ sudo cp fastcgi_params.default fastcgi_params
$ sudo cp mime.types.default mime.types
$ sudo cp nginx.conf.default nginx.conf
$ sudo mkdir conf.d sites-available sites-enabled ssl

Once installed, Nginx requires a little bit of work to hook up to PHP and particularly to work well with Magento.

$ sudo vim nginx.conf  
# Insert the following towards the bottom of the file (but inside the http block) 
map $scheme $fastcgi_https {
   default off;
   https on;
}

##
# Virtual Host Configs
##
include conf.d/*.conf;
include sites-enabled/*;

For each app just add a server block to sites-available, then symlink it to sites-enabled.

$ sudo vim sites-available/magento.dev.conf
# ...     
$ cd sites-enabled
$ sudo ln -s ../sites-available/magento.dev.conf 001-magento.dev.conf

This is the server block definition I use for magento development, feel free to modify it for your needs.

server {
    listen 80;
    listen 443 ssl;
    
    ssl_certificate     ssl/magento.dev.crt;
    ssl_certificate_key ssl/magento.dev.key;

    server_name magento.dev;
    root /Users/aaron/Sites/magento;

    location / {
        index index.html index.php; ## Allow a static html file to be shown first
        try_files $uri $uri/ @handler; ## If missing pass the URI to Magento's front handler
        expires 30d; ## Assume all files are cachable
    }

    ## These locations would be hidden by .htaccess normally
    location /app/                { deny all; }
    location /includes/           { deny all; }
    location /lib/                { deny all; }
    location /media/downloadable/ { deny all; }
    location /pkginfo/            { deny all; }
    location /report/config.xml   { deny all; }
    location /var/                { deny all; }
    location /shell/              { deny all; }

    ## Disable .htaccess and other hidden files
    location ~ /\. {
        deny all;
        access_log off;
        log_not_found off;
    }

    location ~ \.php$ { ## Execute PHP scripts
        if (!-e $request_filename) { rewrite / /index.php last; } ## Catch 404s that try_files miss

        expires        off; ## Do not cache dynamic content
        fastcgi_intercept_errors on;
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_param  HTTPS $fastcgi_https;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        fastcgi_param  MAGE_RUN_CODE default; ## Store code is defined in administration > Configuration > Manage Stores
        fastcgi_param  MAGE_RUN_TYPE store;
        proxy_read_timeout 120;
        proxy_connect_timeout 120;
        include        fastcgi_params; ## See /etc/nginx/fastcgi_params
    }
    
    location @handler { ## Magento uses a common front handler
        rewrite / /index.php;
    }
}

We've said our application lives on a server called 'magento.dev'. So let's tell our hosts file about that.

$ vim /etc/hosts
# Insert or append to an existing line
# 127.0.0.1 localhost magento.dev

Last thing that needs to be done is setting up a selfsigned ssl certificate / key pair and storing them under /opt/local/etc/nginx/ssl

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myserver.key -out myserver.crt
$ sudo mv myserver.key /opt/local/etc/nginx/etc/ssl/magento.dev.key
$ sudo mv myserver.crt /opt/local/etc/nginx/etc/ssl/magento.dev.crt

Once that's done, we can start nginx.

$ sudo port load nginx

Web App Directory Config

I keep my web apps living under /Users/aaron/Sites, but remember that every directory element in the path needs to have the executable bit set for all users (so the web server can traverse the directory tree). Literally this is a case of:

$ chmod a+x /Users/aaron && chmod a+x /Users/aaron/Sites

Install Magento and TAF

N98 Magerun is the coolest thing to happen to Magento development since well, I can't remember. It singlehandedly relegated a few thousand lines of cobbled together bash script to the bin.

$ cd /Users/aaron/Sites
$ curl -O magerun.phar https://github.com/netz98/n98-magerun/raw/master/n98-magerun.phar
$ chmod a+x magerun.phar
$ ./magerun.phar install
# Follow the directions and install to /Users/aaron/Sites/magento with base url http://magento.dev and database name 'magento'.

After all that work, hitting http://magento.dev should now bring up the magento demo store!

I've been playing with Magento's Test Automation Framework and it was the motivation for finally getting everything working properly natively.

TAF runs at a glacial pace and in my normal development environment (VirtualBox over NFS), the universe would have undergone heat death long before the TAF suite completed its running.

Unfortunately the documentation for TAF is a bit of a mess (I'll write about my experience with it soon), but what it offers - 1500 automated tests - is a pretty big attraction.

Installation is actually pretty easy. I am assuming you don't have git already installed (remember you can use port variants to see what extension options are available):

$ sudo port install git-core +bash_completion +credential_osxkeychain +doc +pcre +python27
$ sudo port install git-extras
$ cd /Users/aaron/Sites
$ git clone https://github.com/magento/taf taf
$ cd taf # /Users/aaron/Sites/taf
$ cp phpunit.xml.dist phpunit.xml
$ cp config/config.yml.dist config/config.yml
$ cd .. # /Users/aaron/Sites
$ curl -O selenium-server.jar http://selenium.googlecode.com/files/selenium-server-standalone-2.31.0.jar

To run the test suite open up a new terminal

$ cd /Users/aaron/Sites
$ java -jar selenium-server.jar

Now the test suite is good to go

$ cd /Users/aaron/Sites/taf
$ ./runtests.sh

The test suite takes a loooooong time, so go for a run or something.

Hopefully these steps help out other PHP developers suffering from OSX.

SQLSTATE[HY000] [2002] No such file or directory.

You fix it by just appending the sock file for the mysql version you're using into the PHP mysql.ini file. I use mysql55 so to fix PHP I do this

echo 'pdo_mysql.default_socket=/opt/local/var/run/mysql55/mysqld.sock' | sudo tee /opt/local/var/db/mysql.ini

$ sshfs -o ssh_command="ssh -i ~/ssh_keys/aaron@awshost.pem" aaron@aws.instance.com:/var/www/ ~/Sites/awshost

Actually, right after I posted this, I realised there's a better way to do it. That is, use the 'IdentityFile' option instead (as per the format of .ssh/config).

$ sshfs -o "IdentityFile=~/ssh_keys/aaron@awshost.pem" aaron@aws.instance.com:/var/www/ ~/Sites/awshost

If you have any problems then add '-o debug' to the above command to help track it down.

There are only two hard things in Computer Science: cache invalidation and naming things.

-- Phil Karlton

I wanted to grab the last bit of a url that I knew would be the name of an image. I knew strstr well but that operates by giving you the remainder of a string that occurs after some needle in a string haystack. I wanted this behaviour, but only from the last instance of the needle.

strstr — Returns part of haystack string starting from and including the first occurrence of needle to the end of haystack.

strrchr -This function returns the portion of haystack which starts at the last occurrence of needle and goes until the end of haystack.

Let's look at how these work with an example

$url = 'http://www.google.com/a/b/c/d,img';
echo strrchr($url, '/'); // prints /d.img  
echo strstr($url, '/');   // prints //www.google.com/a/b/c/d.img

Now I've been programming in PHP for pushing on 12 years and this one still did my head in. The names of two very similar behaving functions bare little resemblance to each other.

At this point the arguments and criticisms over the core API have been exhausted and there's little that can/will be done. But I do wonder if it would be worth creating an object library to encapsulate primitive functions such as String, Integer, Array, Float etc., I'm not sure how possible auto-boxing is with PHP, and indeed if it's even a good idea. But definitely some object wrappers would help ease this API pain.

The best way to predict the future is to invent it. -- Alan Kay

Recently I have written a little bit about Smalltalk, and in my enthusiasm I got hold of a book called Dealers of Lightning by Michael Hiltzik. It covers the rise and fall of Xerox's Palo Alto Research Center (PARC), the research center from which Smalltalk emerged.

I initially read it to learn more about the context in which Alan Kay imagined Smalltalk and to find out who was the Executive-X he mentioned in The Early History of Smalltalk (it was Jerry Elkind). However I ended up coming away with a lot more. In particular, a new appreciation for a number of scientists I previously knew very little about. Scientists that are almost single handedly responsible for the shape of modern computing.

I grew up in the 80s so I have no real personal appreciation for computing as it was before say, 1984. I had an IBM clone (an Amstrad) and a VIC20. At school we have Commodore 64s/128s to play with. So for me, a computer has always been something that sits on your desk, you turn it on and you type away and stuff pops up on the screen. But right up until the late 70s this paradigm was considered absurd. Wasteful even. It took the vision of Alan Kay and the technical genius of Chuck Thacker and Butler Lampson along with the almost unlimited cash of Xerox to realise.

The book opens up by transporting the reader back in time to the late 60s and lays out the genesis of PARC. It then proceeds in roughly chronological order with each chapter focusing on one of the scientists and/or their inventions. The book closes by looking at some of the reasons why Xerox couldn't transform its research into viable products. Nominally the story is about what Xerox PARC did, however Hiltzik couches everything in terms of the scientists and it is his ability to bring these characters to life that makes the book so riveting to read.

One of the most striking individuals of the story is the Impressario Robert Taylor, a man who as much as anything can be considered the grandfather of the Internet (nee ARPANET). The Kays, Thackers and Lampsons of the story are the geniuses but genius needs direction and at times support. This is the role Bob Taylor played. The story of PARC for better and worse revolves around him and his relationship with the researchers that shared his vision of interactive computing and those whether in his or the other labs, or in management, that well didn't.

The Computer Science Lab (CSL) was a collection of engineers who weighed everything pitilessly against the question: How will this get us closer to our goal? They had commited themselves to developing Xerox's Office of the Future and anything that diverted their attention or served an alternative goal had to be discarded or obliterated.

It is Taylor's utterly single minded vision of interactive computing that drives much of the success and much of the drama of PARC. Taylor was in continual combat with the other labs for resources and funding and inevitably with his managers George Pake, Jerry Elkind and eventually Bill Spencer. But that was outside his lab. In it, he was the oil that kept the cogs turning and among his staff he was considered a unique and brilliant manager of researchers. It is on this skeleton of contradiction and conflict that the guts of the story of PARC hangs.

The book contains a number of particularly powerful scenes. Two particularly stuck out for me. The first is Alan Kay, his vision for a Personal Computer brusquely put down by CSL Manager Jerry Elkind, falling into a depression. Alan Kay is well known for his brilliance and verbal flourish but Hiltzik does well to also bring home his vulnerability in a way a modern reader would not expect. Kay would ultimately realise his vision of a Personal Computer - with the help of Taylor's CSL - while Elkind was seconded away from PARC on a Xerox taskforce. We tend to recall Kay's assertive (and largely proven) views on Computing. It is unexpected and moving then, particularly with the benefit of hindsight, to see him doubt himself and his ideas before they were fully realised.

The other scene involves Adele Goldberg, co-developer of Smalltalk, and her reactions to Apple's infamous raid on PARC. If you've ever seen The Pirates of Silicon Valley you might have a feel for how this all went down. But Hiltzik's account of it conveys such a sense of dread and hopeless frustration that the movie never came close to recreating.

By the end of the book Taylor's time at PARC draws to a close and with his departure so too does the most storied era of PARC. Scarcely six months after Taylor's forced resignation, the majority of his lab also resign and either follow him to Digital Equipment Corporation (creators of the famous PDP series of minicomputers), or join one of the many startups blooming in Silicon Valley following the success of IBM and Apple's Personal Computer products.

Dealers is fundamentally a story about people that just happen to be in technology - rather than a book about technology itself. It is a human story. It is about what happens when you take the cream of a generation's scientific talent put them in one place and throw lots of money at them. It is about what happens when you combine a visionary maverick with the proneness to credentialism by academically minded administrators. It is about what happens when you have corporate management that want to embrace change but either do not understand it, or worse, fear it.

It is the book's focus on the people of Xerox and PARC particularly, their feelings, motivations and backgrounds that brings this extraordinary tale of modern computing's birth to life.

Google Checkout: Error parsing XML; message from parser is: cvc-datatype-valid.1.2.1: '' is not a valid value for 'decimal'.

I have developed custom modules which add free or bonus items to a customer's cart if they use coupons, meet certain cart criteria or belong to particular customer groups. Buy x, get y rules also work this way. So this is a nuisance. Luckily few customers opt to use GoogleCheckout, but still, I don’t Live with Broken Windows[1].

Chasing the problem down the call stack leads to app/code/core/Mage/GoogleCheckout/Model/Api/Xml/Checkout.php and specifically the _getItemsXml() method.

$unitPrice = $item->getBaseCalculationPrice();
if (Mage::helper('weee')->includeInSubtotal()) {
    $unitPrice += $item->getBaseWeeeTaxAppliedAmount();
}
// ...
<unit-price currency="{$this->getCurrency()}">{$unitPrice}</unit-price>

Now, if the product's baseprice is 0, then for some unfathomable reason it's set to '', not 0. As the unit-price element expects a decimal value, an empty string fails validation.

The fix is pretty trivial

$unitPrice = $item->getBaseCalculationPrice();
if (Mage::helper('weee')->includeInSubtotal()) {
    $unitPrice += $item->getBaseWeeeTaxAppliedAmount();
}

$unitPrice = ((float) $unitPrice > 0) ? $unitPrice : 0.00;

The store I needed to fix only used US dollars so I haven't tested how the use of other currencies or locales might affect this fix.

To apply the fix, don't modify the core codepool, but instead take advantage of the local and community codepool's higher classloader priority[2] and place the amended code in app/code/local/Mage/GoogleCheckout/Model/Api/Xml/Checkout.php.

[1]: 'Don't Live With Broken Windows' is a tip I first read about in The Pragmatic Programmer. It is used to help fight Software Entropy (software's tendency to lose structure over time). This concept has parallels with the real world as urban areas with broken windows tend to see higher levels of vandalism when compared to areas where windows are constantly maintained.

When you ignore small problems it becomes easier to let more significant problems slide too. Hence the rule of thumb, 'Dont Live With Broken Windows'.

[2]: Magento resolves classes in this order local, community then core. This means if two classes have the name Mage_Core_Model_Foo one exists in local the other in core, then the version in local is used.

Thankfully there's a really simple way to socialise Nginx. It is by calling the nginx server command directly with the -s argument instead of using the /etc/init.d/nginx or service nginx commands.

-s lets you send signals to the Nginx master process and Nginx behaves differently whether it receives a quit signal versus a term signal.

# terminate the nginx master process immediately
$ sudo nginx -s stop 
# terminate the nginx master process once all outstanding connections have been completed
$ sudo nginx -s quit 

$ abonner@avalanche:~$ ps aux | grep nginx
root      1063  0.0  0.0  88796  3432 ?        Ss   Jan21   0:00 nginx: master process /usr/sbin/nginx
www-data  9786  1.3  0.0  91564  7352 ?        S    Feb03 190:11 nginx: worker process is shutting down
www-data  9788  1.3  0.0  91288  7072 ?        S    Feb03 189:02 nginx: worker process is shutting down
www-data  9789  1.3  0.0  91160  6956 ?        S    Feb03 190:03 nginx: worker process is shutting down

Let you visitors finish their drink, don't terminate nginx on a production server using /etc/init.d/nginx stop or service nginx stop.

Read more about Nginx's command line options

This year I didn't bother with any. I did in November though, resolve to read more technical books, and to particularly focus on the 'classics'. My motivation was stirred by Panagiotis Louridas's essay 'Rereading the Classics' from the book Beautiful Architecture. In it, Louridas examines the structure of Smalltalk and tries to reason why it achieved greater lasting success blazing a trail for others to follow than as a practical working programming environment.

Louridas suggests that Smalltalk is a classic and by way of justifying why, he quotes Italo Calvino's Why Read the Classics (1986)

The classics [...] exert a peculiar influence, both when they refuse to be eradicated from the mind and when they conceal themselves in the folds of memory, camouflaging themselves as the collective or individual unconscious.

A classic does not necessarily teach us anything we did not know before. In a classic we sometimes discover something we have always known (or thought we knew), but without knowing that this author said it first, or at least is associated with it in a special way. And this, too, is a surprise that gives much pleasure, such as we always gain from the discovery of an origin, a relationship, an affinity.

The classics are books which, upon reading, we find even fresher, more unexpected, and more marvelous than we had thought from hearing about them.

A classic is a book that comes before other classics; but anyone who has read the others first, and then reads this one, instantly recognizes its place in the family tree.

Smalltalk is a language that most of us have heard about but have rarely seen, and after being introduced to it I was inspired to start digging up any further literature I could find. The more that I read, the more that I appreciated its cleverness. Smalltalk can be eerily familiar, and as you begin to grok its syntax it is easy to recognise aspects that have inspired certain features of our 'modern' OO languages.

It is striking that the language itself (and indeed much of the material written about it) is so old. The most modern version of Smalltalk is Smalltalk-80 and although there are modern implementations, the overwhelming bulk of the syntax and environment still cohere to the 1980 standard. Yet download Pharo, read Alan Kay's The Early History of Smalltalk (1993), or Dan Ingalls's Design Principles behind Smalltalk (1981) and it all still seems contemporary. The language itself, which was controversial at the time for eschewing ALGOL syntax, has aged very well. Ruby has maybe made them fashionable, but Smalltalk sported things like block closures over a quarter of a century ago. Reflection, Metaprogramming and dynamic typing - all in Smalltalk in the 70s. Even the idea of using virtual machines to host the programming and operational environment seems remarkably contemporary today, as we increasingly move to abstract our programs and programming environments from bare metal.

It is humbling to see so many ideas that we take for granted today already implemented on a platform that is over a quarter of a century old. It is like amazing at how the Egyptians built the pyramids. Sure we could do it now, no sweat. But we have so much more raw engineering knowledge to throw at the problem. Alan Kay, Dan Ingalls, Adele Goldberg and the rest of the team at the Xerox PARC Learning Research Group designed and implemented Smalltalk with less computing power than my fridge has today. Despite formalising most of the vocabulary for OO software development during the development of Smalltalk it's hard not to feel like some of the energy and innovation of Kay's thinking didn't survive the C++ and Java succession.

Our discipline is still quite young when compared to traditional engineering or the sciences. Yet we seem to keep facing the same problems over and over again. The only difference is perhaps a few orders of abstraction, bigger piles of data and slightly more exotic technologies. But when I think that the fundamental concept of 'Agile' or at least 'iterative' development was doing the rounds in the 60s, it made me wonder what other insights are out there, buried in the forgotten past.

Maybe one reason we tend to forget what others have learned is that the average developer only reads one technical book per year. That means a sizeable percentage of software professionals do not actually read any! I think if I average over my professional working career (after graduating at the end of 2004) I would probably be batting one per year too. If I were to look over the last three years, maybe two per year. I wonder how many mistakes I might have avoided in that period had I read Fred Brooks's Mythical Man Month in 2004 rather than 2012.

My excuse has always been a lack of quality time. Last year in July I left my fulltime job for the world of consulting/freelancing. I imagined it would be easier to read more. In practice it has been, but not as easy as I thought or hoped.

I have a young daughter, my partner has returned to work and despite working from home, I still put in 50-60 hour weeks. It doesn't leave a lot of spare time and what spare time there is, is usually late at night when it's hard to concentrate.

So since I made a determined effort to read more, I've read four books cover to cover, and cherry picked bits out of another four. It feels good and the trick I've found is to read a chapter at a time, whenever you can. Whether it's just before dinner, over coffee/lunch, waiting at the supermarket or just before bed. I found that by reading, even if was just a little bit, every day, I was starting to get through entire books.

It requires conscious effort though, and some material is more suited to this style of reading than others. After a long day just before bed it's pointless trying to delve into SICP. It helps to read certain books during the work day, for example I've been re-reading GOOS and going through Kent Beck's TDD by Example over lunch times. I actively practice TDD so reading a chapter from one of these books midday helps me relate it directly to what I'm working on when I go back into the office.

I have a huge reading list setup in Google Reader, and I'm starting to think it's distracting. I have a slight OCD in that I need to keep the unread count at zero. At the end of a thirty minute work sprint, I would take five minutes to quickly flick through the list. It is distracting actually, and most of the content is superficial. The benefit of reading a book over a blog is the tendancy for a book to have its ideas more fully formed and logically structured. I realise the irony saying this while writing a blog myself. I feel that blogs have their place, but I have been spending far more time reading blogs than reading books. I am now leaving my Google Reader list unread longer and after a few 30 minute sprints picking up a book instead.

As I wrote above, Louridas inspired me to learn more about Smalltalk, and any research into the language leads to Alan Kay's ACM paper The Early History of Smalltalk. The paper beyond providing a wonderful insight into the language and the Xerox PARC is the source of some great quotes. One of my favourites goes:

Where Newton said he saw further by standing on the shoulders of giants, computer scientists all too often stand on each other’s toes.

In Software Engineering, we are so busy looking forward that we don't look back often enough. There is such a rich wealth of knowledge out there already considered and documented. I think we all should make more of an effort to re-discover it.

Here's a small selection I have picked up recently that I didn't know.

sudo !! 

Run the previous command as sudo. This is great when you realise you needed to run something as root.

ctrl-x e

Open up $EDITOR to enter in a long command. In my setup it fires up vim. This is great for some of the long rails commands you need to create controllers or models.

cat /etc/passwd | column -s':' -t 

Column, columnates input, the -t argument will format standard input into a table and -s lets you specify an arbitrary field delimiter. For unformatted input this is very handy.

These next few are specific to zsh, and while I do love bash, since switching to zsh I haven't really looked back. It's things like this that when you work with a terminal every single day, you can't give up.

aaron@tempest ~ $ d                                 
0	~
aaron@tempest ~ $ cd /etc
aaron@tempest /etc $ d
0	/etc
1	~
aaron@tempest /etc $ 1
~
aaron@tempest ~ $  

The 'd' command lists the directory stack, and then entering an integer will switch you directly to the directory index in the stack. It is a killer app.

Moving directories also is very pleasant in zsh. Use '..' to move up a directory, and simply type the name of the directory in, to move into a directory.

aaron@tempest ~ $ ..       
aaron@tempest /Users $ aaron
aaron@tempest ~ $ 

This last one is a trick I've know for a few years, I don't know how much time this has saved me exactly, but I use it every single day.

In vim, if you're editing a file that requires root (or any other user) permissions, you can write the file by doing

:w !sudo tee %

I use it so much that I've set up a leader key binding in my .vimrc

nnoremap <leader>sr :w !sudo tee %<CR>

There's nothing more annoying than making lengthy changes to a config file, go to write it and getting permission denied...

I make all my configs available online at github, if you're interested in seeing how I setup my environment.

To cut a short story, very short, you do this (at least if you have macports in /opt/local (the default), and are using the mysql55 package).

$ gem install mysql2 -- --with-mysql-lib=/opt/local/lib/mysql55/mysql --with-mysql-include=/opt/local/include/mysql55/mysql 

This matters a lot when you want to use xdebug on a project that runs off a virtual machine, or even perhaps a remote server.

The typical pattern (when using PHPStorm in my case) to invoke a remote cli debugging session is to set an environment variable telling the IDE what server configuration to use, and to tell PHP what remote host to connect to.

$ PHP_IDE_CONFIG='serverName=mydevmachine.local' php -dxdebug.remote_host=192.168.0.1 myphpscript.php

Now this will work fine, however if we want to debug during a phpunit test normally you would do this

$ PHP_IDE_CONFIG='serverName=mydevmachine.local' phpunit -dxdebug.remote_host=192.168.0.1 -c phpunit.xml

Unfortunately this doesn't appear to work at the moment (version 3.7.9). If I use the xdebug test client, I can see xdebug trying to connect to the localhost, ignoring what I've told PHPUnit. I'll look into this a bit more later, but I suspect PHPUnit isn't passing on the php.ini settings in a timely fashion for xdebug to hook into.

The solution to this problem is to make use of ssh port forwarding. This works exactly the same for a virtual machine as it would for a remote host, which makes xdebugging on a production machine (hopefully only ever in an emergency!!!) much more straight forward (and less insecure).

$ ssh -R 9000:localhost:9000 myvm.local

This sets up myvm.local to forward all connections to its localhost on port 9000 to the remote client's port 9000. When xdebug goes to connect to localhost:9000, it ends up actually connecting to mydevmachine.local:9000.

It's a bit of a hack, but a time saving one. The other alternative is Vim and its xdebug plugin. This isn't a bad alternative. But once you've experienced the power of PHPStorm's debugging implementation it's hard to go back.

It has been an exciting time to be a PHP Developer these past twelve months, PHP 5.3 is now rock solid and PHP 5.4 is getting there. Both releases significantly modernise elements of the language, closing the gap between PHP and the offerings of more 'in vogue' languages.

In technology we often see change happen in sudden, explosive steps. Often it seems to coincide with developments a technology's ecosystem or among its competitors. For PHP the first major kick was the rapid rise in popularity of Object Oriented Programming in the early 00s. This led to PHP 5's radically overhauled OO implementation in 2004. The next kick, I feel, came in 2005 when Ruby on Rails exploded into everyone's consciousness. RoR provided a full stack web development platform that drastically simplified creating complex web applications. The PHP community responded in kind with a number of 'fullfat' Model View Controller (MVC) Frameworks, the most successful being Zend's and Symfony.

It all depends

The arrival of PHP 5.3 and features like Namespaces, PHAR, Closures and the ubiquity of Github is having the effect of giving PHP a new kick, and the results are starting to make themselves felt. We now have second generation frameworks from Zend and Symfony leveraging these technologies.

One problem remains though, and that is managing and distributing dependencies. Modern web development platforms all now have robust dependency management tools available and in the PHP camp, PEAR wasn't really cutting it.

The success of Symfony2 in particular, with its emphasis on high quality, modular components, forced PHP developers to address how they bundled and distributed library code.

Luckily for us, the guys behind Composer, (again, taking considerable cues from the Ruby community) have licked it. Composer, in tandem with Symfony2 components allow PHP Developers to confidently build on top of other developers' libraries.

Do we really need another packaging tool?

Why did we need another package and dependency management tool anyway? What really, is wrong with PEAR? Well, if we wind the clock way back to 1999 when Netscape Communicator was still the most popular web browser and Google had just moved out of Susan Wojcicki's garage, PEAR was conceived as PHP's answer to PERL's CPAN. Despite some strident efforts, it never really managed to become the most pleasant package manager to work with: rigid, elitist and worst of all, difficult for end-users. PEAR's age strictly speaking is not the problem, but its centralised nature is a bottleneck and there is no straightforward way to handle two packages with varying dependencies. For example: say package x, requires stable package y. Package z requires beta package y. You can't install both. Dependency and package management has moved on a long way since 1999.

Over time PEAR's shortcomings have led to a graveyard of abandoned packages, code of at best variable and at worst, dubious, quality, and a community lacking in any sort of dynamism. If you make something easy, people will use it. PEAR is difficult to use for developers and users alike.

Composer democratises (in the best sense) things and puts full control of dependencies in the hands of library developers. Free to pick and chose code they want to use. Free from having to worry about navigating the PEAR jungle. Here the rise and rise of Github has been key. Composer can sit over the top of code distribution services provided by Github, or it can use its default Packagist repository. This removes the need for libraries to live in a blessed canonical repository or for developers to host it themselves.

... profit?

There's no compelling need now to constantly rewrite basic library components (I think we've finally licked what ought to be the basic issue of class loading!), Free of the shackles of PEAR, we are witnessing an explosion of high quality PHP frameworks, libraries and utilities.

PHPSpec, Behat, Twig, Mockery, Doctrine, are just a few that immediately spring to mind. Some (such as Doctrine) have been around a while. However the advances PHP 5.3 brought to the table have significantly helped improve the utility of these projects.

Anyway, so (after a fashion) I come to the tool that motivated me to write this post, n98-magerun.

The name is horrible, but the tool itself is brilliant. In short, it's Drush for Magento and it's wonderful. It is one of those tools that makes you wonder what on earth you did before it.

I have a folder full of bash scripts, cobbled together to help automate the mind-numbing process of managing Magento installations. Over the course of a few months Christian Münch and friends have overseen a small tool quickly develop into the kind of utility we've all wanted but never had the time/patience to build ourselves.

Magerun is elegantly simple for the user, and cleanly extendable by developers. It is a perfect illustration of why it's such a great time to be a PHP developer. Better dependency management, easy distribution, modular libraries and powerful language syntax have all came together to let someone with an itch, scratch it quickly and effectively.

It has become several orders of magnitude easier to develop, package and distribute PHP libraries and utilities. The result of this leap forward is a brilliant tool that helps Magento developers dramatically increase their productivity.

I'll be writing about how awesome Magerun is shortly, but just one of its cool features is the ability to dump out a merged version of Magento's config.

This is extremely helpful when trying to resolve conflicts between modules, or figure out what bit of configuration is taking precedence.

The resulting xml though is pretty raw and unformatted, but xmllint can fix that.

Xmllint expects a file to work with and cannot use piped input. So we need to use bash's Process Substitution feature to avoid having to create temporary files.

$ xmllint --format <(magerun config:dump)

So, magerun and xmllint, a simple way to get a formatted, easy to examine view of how Magento is putting your install's configuration together.

$ git show --name-only <commit>

This will give you a list of affected files in commit

Alternatively if you don't care what differs in the specific contents between two branches, and only want to see different files you can do

$ git diff master..origin/master --name-only

This will show you the list of files that are different between your local master branch and the remote master branch. Handy if you have just done a git fetch and want to see what's different before merging or rebasing.

First thing, shut down the running instance and then restart it directly

$ sudo -u <mysql_user> mysqld_safe --skip-grant-tables --skip-networking

The --skip-networking option is important, as by skipping the grant tables, any user can connect to the running mysqld service, will full permissions.

Once you've started the server up, login without a password, and issue an update query to the mysql.user table.

$ mysql -uroot mysql
mysql> UPDATE user SET password=password('newpassword') WHERE User = 'root'

Close down mysqld and restart. You're good to go.

We've seen some pretty epic PHP rants this year, probably the most famous among them are PHP a Fractal of Bad Design, and Jeff Atwood's latest (in what seems to be a biennial broadside) The PHP Singularity.

The common thread in these rants is incredulity that anyone would, in 2012, write new code in PHP. There's a lot of reasons why someone might write greenfield PHP code in 2012. But equally (and this is said as a decade long PHP programmer) I have to admit that plenty of the criticisms levelled at PHP are valid. Yet, for the most part they just don't, particularly, matter.

One criticism that is wholly invalid yet comes up time and time again, is that using PHP intrinsically leads to bad code. I don't feel this is an inherent trait of PHP itself, so much symptomatic of the popularity and low barrier to entry of PHP. Basically there's more examples of bad code out there compared to pretty much any other platform because there's simply more code out there, written by programmers of wildly varying skill. The other problem, is that PHP came into existence as a scratch to a C programmer's itch. PHP was developed at a time where people still actually wrote web applications in C and where the stateless nature of HTTP was relatively respected. PHP, like the web itself, has moved on dramatically since then.

A modern PHP 5.4 webapp looks about as similar to an early 00s PHP 4 webapp as Scala does to Java. Yet many critics when slating the language appear to be code archeologists, excavating pre-historic practices that went out of favour long-ago.

This can be partially forgiven, because owing to the age of the language there's plenty of out of date information out there with high rankings in Google. The ubiquitous w3schools is an unfortunate example of bad practices coming well ahead of sites with more modern approaches to solving problems in PHP.

So 'New PHP' is very different to 'Old PHP'. But the 'Old PHP' is what most people seem to find when searching in Google and this confuses people.

We see this manifested in blogs like Will Rails become the new PHP. This blog has a number of spectacular shortcomings, most egregiously the author's horribly naive view of the PHP community, but the interesting one is his ignorance of the power of Google.

There's plenty of support out there for budding PHP programmers, whether on the web, on forums, or IRC. There are countless, well attended, supported and growing conferences, meetups and the like for PHP programmers. One thing that a community cannot do, is force Google to nuke w3schools' PageRank. Which means that brilliant efforts like PHP The Right Way get swamped by old, incorrect and at times dangerous dreck.

And what the 'Will Rails become the new PHP' author perhaps hasn't realised, is Rails, at least in the terms he's trying to couch it, has already become the 'New PHP'. I am a novice Rails developer, I like to hack around in it as it can be quite fun to spike out solutions. What I've been struck by, is the sheer amount of bad advice out there. Advice novices will come across, if they turn to Google for help.

If you're well versed in a platform you learn through brutal experience what works, what doesn't and your nose is finely tuned to bullshit. When I read a PHP article I know instinctively if what I am reading is reliable. But with Rails, as a novice, I don't quite have that sense, beyond my own background experience with other programming languages.

So let's look at an example. I've been working on a dead simple Rails authentication webservice. It listens for HTTP requests for /login, /logout, /session, etc., and emits either XML or JSON in response. I'm using the respond_to method to serve out these responses. Unfortunately what I found is if I request a route that does not exist, I get an HTML error back. This doesn't make a lot of sense for a webservice that otherwise speaks XML and JSON.

Other global exceptions similarly respond with HTML. I don't want to wrap every action up in a begin/rescue block and there is certainly no way to intercept router exceptions in actions anyway. So I needed to learn how to catch global exceptions.

In my journey of (Google) discovery I came across this blog post and appeared to hit paydirt. The advice appears to be legit, hell someone in the rails community even featured it in a podcast. So on the face of it this seems good. But that switch statement sure is smelly. Does it, really, need to be this hard?

Now my general purpose programming brain recognised that this code while solving my problem, is not ideal. And why is it not ideal? This one method sure has a lot of responsibility. Method names with plurals in them are usually a code smell. Over time as more specific exceptions need to be handled I would end up with code that is as easy to read as Goethe's Faust, photocopied and in the original Gothic script (not easy). What we have here is a God Method in training.

Now, what if I am a Ruby/Rails/Programming novice? I have got plenty of other stuff to learn, I'm going to go right here and Cargo Cult this code into my webapp and move on. Just like all the rookie PHP coders do, right?

Well, I didn't do that. I saw that this rescue_from method was pretty awesome and so I went to the Rails API docs to look it up.

API docs for any language are pretty terse, but what jumped out at me was this line:

“Handlers are inherited. They are searched from right to left, from bottom to top, and up the hierarchy. The handler of the first class for which exception.is_a?(klass) holds true is the one invoked, if any.”

This isn't great documentation admittedly, but it means basically, if you put rescue_from Exception at the bottom of the list of rescue_from handlers in your application_controller.rb file, then since everything derives from Exception, nothing else will get a look in (Rails will look at the handlers from the bottom up).The author of that helpful blog we found didn't realise this, and so his solution was needlessly complicated.

What can we learn from this? Well Rails programmers certainly live in a glass house and shouldn't throw stones is one thing. But on a slightly less trollish note, there is a problem here for all novice programmers that turn to Google to help them solve problems. The answers on Google are usually either wrong, or at best, incomplete. As the web gets older, bad and out of date advice piles up making it much harder for novices to find good advice.

Knocking a language for this phenomenon (or a framework, seriously, whatever) is more than a little ignorant and doesn't solve the problem. Efforts like PHP The Right Way is how PHP is trying to fix it. If Rails really doesn't want to be the 'Old PHP', they need to realise it's less to do with languages and platforms, and more about SEO.

To see what branches your local repo things exists you do something like this:

$ git branch -rv
> origin/1620-upgrade  2e0cc56 Ignore active local.xml from vc
> origin/HEAD          -> origin/master
> origin/cas-sso       2351be5 Add gateway logiin and logout support
> origin/giveaways   63daf5a Use cms blocks for banner placements
> origin/master        496c975 Merge affiliate module
> origin/newskin      d7220c9 Optimise skin and ui images
> origin/release       496c975 Merge affiliate module

So this is my local Magento git repository, many of the branches here are now defunct and no longer in the remote (i.e. I had previously had used $ git push origin :branch from another host)

To refresh then I need to prune my branches list. The git incantation to do this is

$ git remote prune origin
> Pruning origin
> URL: dev@vcs:git/store.git
* [pruned] origin/1620-upgrade
* [pruned] origin/giveaways
* [pruned] origin/newskin

Looking at the remote branch list again:

$ git branch -rv
> origin/HEAD    -> origin/master
> origin/cas-sso 2351be5 Add gateway logiin and logout support
> origin/master  496c975 Merge affiliate module
> origin/release 496c975 Merge affiliate module

A simple configuration recipe for adding new cache tags to the Magento backend’s “clear cache” feature.

In theory the Magento cron subsystem should keep a lid on these tables growing too big, but I avoid using Magento cron, preferring to handle that myself directly via crontab tasks.

The other option is to write your own table cleaning script (or copy one from somewhere), and this will work too. But it's annoying, if you don't want this log data, why write it in the first place.

So my solution is to disable it by removing the observer events that perform the logging.

I have this in my local.xml which takes precedence over other nodes in the config and therefore overwrites them. Here, by setting the observer to be the string 'disabled', the existing observer event is removed and replaced with something that will never be fired.

Now, you don't need to worry about periodically cleaning out your database, nor do you need to fear a 3am text message from your production DB servers screaming about the disk being full...

I'm going through some PCI scan results this morning, and in the main it's going well, but I got a couple XSS hits on our catalogsearch pages. This is odd, I think. I've audited these pages, they definitely get routed through magento's escaping code.

On closer examination it turned out the form was okay, it was via the breadcrumbs, that unescaped input was getting into the wild.

I'm running Mage 1.6.x so this code may look a little different if you're running 1.7

Take a look at app/code/core/Mage/CatalogSearch/Block/Result.php, and specifically at the prepareLayout() method:

Now if you look at line 11, if breadcrumbs are enabled, unescaped input is happily added ready for output.

$title = $this->__("Search results for: '%s'", $this->helper('catalogsearch')->getQueryText());

This fix is easy, replace line 10 with:

$title = $this->__("Search results for: '%s'", $this->helper('catalogsearch')->getEscapedQueryText());

This is a really neat example of the evils of duplication and where bad programming practice can lead to real world problems. I am speculating, but it seems reasonable to infer that the original programmer got trigger happy with the copy & paste keys. Later, at some point you could imagine another engineer coming in to XSS safe the code fixed one bit, but (and programmers are human) missed the other (exactly the same line), and we end up with an issue like this.

Personally, I patched the file as described above and stuck it in app/code/local/Mage to override the core code pool version.

Unfortunately this doesn't happen very often and when using a stack of Rubygems, you almost always get bitten by something.

My cause for complaint today is Vagrant and Chef, well specifically Chef Solo. Vagrant is fine, it does what you tell it to do, but for most use-cases Chef Solo is the right tool to use for provisioning your virtual server. The Vagrant docs on Chef Solo unfortunately fib you, they say you can use Data Bags with Chef Solo, but by default you cannot.

This is a big deal as many useful Chef recipes make heavy use of Data Bags. Data Bags which let you provide environment specific configuration for your provisioning is not yet supported by the stock Chef Gem (currently version 10.12.0). In order to make use of Data Bags with Chef Solo, you need version 10.14.0 and above. This means building the gem from source.

I use Veewee to build my vagrant base boxes (you should too, it's awesome!), and you can edit the postinstall.sh file in your box definition folder to build Chef from source, rather than installing it directly via Rubygems.

You can repeat this for your local dev machine, and now you can get Chef Solo cooking up your recipes and happily using data bags.

This overrides the core code pool's adminhtml definition, and puts a dependency on a non-existent module. Effectively, this disabled the menu item because it no longer meets the defined dependency requirements.

As always with any magento configuration / module changes, you may need to clear caches for this to take effect.

Yesterday I had to do an emergency restart of mysql in production and the resulting magento report/xxxx files swamped out everything else that I might have wanted to look at.

So specifically, I wanted to delete all the files that were created between a start and end date.

GNU find makes this easy

$ find . -type f -newer 655958024588 ! -newer 627812220533 -ls

This instructs find to list (-ls) all files (-type f) that are newer than a file called 655958024588 (-newer) and not newer than 627812220533 (! -newer).

If you do not have two files to act as date range boundaries, you can use touch to create them.

$ touch -t yyyymmddHHMM start_date_file
$ touch -t yyyymmddHHMM end_date_file

Then supply these file names to -newer and ! -newer.

To delete the files we can use -exec.

$ find . -type f -newer 655958024588 ! -newer 627812220533 -exec rm -rf {} \;

Here it's the -exec argument does the heavy lifting. {} is a placeholder for the file name (find substitutes '{}' with each found filename) and ; terminates the command sequence (much like it does in regular bash).

So, if you want to use veewee to setup a new amd64 Precise Pangolin basebox for vagrant, you either have to pull the latest veewee sources from github, or download the most recent releases templates and copy them over into your veewee gem folder.

This is where veewee-templates-update steps in, it automates that latter step (downloading and installing just the updated templates) for you.

Installation is simple:

$ gem install veewee-templates-update

Then, just run the updater:

$ veewee-templates-update
> Veewee: /home/aaron/.rvm/gems/ruby-1.9.3-p194/gems/veewee-0.2.3
> Downloading: https://github.com/jedi4ever/veewee/tarball/master
> Extracting: CentOS-4.8-i386 CentOS-5.5-i386-netboot CentOS-5.5-x86_64-netboot 
> ...

$ vagrant basebox define precise-amd64 ubuntu-12.04-server-amd64
> ...
> (profit)

Anyway, quite happy with it so far, but like any geek with new kit, I want to know everything about it, and make it dance to my whim.

Two things learned this morning: if you have an old Magsafe (1) powerpack, which I did from my old 13" Mac Pro, and it has higher or equally rated wattage you can use it with your macbook air.

It makes sense, a high rated powersupply can support a lower power rated device, but not vice versa. So a 60 watt Macbook Pro magsafe can power a 45 watt Macbook Air. But a 60 Watt Magsafe can't power a 15" 85 watt macbook pro.

The other thing which I have discovered, is that the thunderbolt port on the newer Airs use the same socket form factor as the mini display port. This means is if you have an old Mini DisplayPort -> DVI adapter lying around, you can reuse it.

My first real job has been to get a decent unix environment up. OSX's BSD utilities don't really cut it. Macports is far and away the best distribution out there.

Once you install coreutils and get ls, find etc it makes sense you will want to change your shell to a modern version of bash (or zsh if that's the way you roll).

Lion ships with bash 3.2 whereas Macports will give you a contemporary version 4.2 Unfortunately it's not as simple as going

$ sudo port install bash
$ chsh 
  <input /opt/local/bin/bash>

I had to do this before, but I'd forgotten there's a trick to changing your shell in OSX to a non-standard location. The file /etc/shells contains a list of valid shells chsh will permit. You need to edit (as root or via sudo) this file and add your macports shells. Once that's done chsh will let you change no problem.

To take a column from a MySQL database and quickly output it ready formatted as a Javascript array literal (without any specific escaping) do:

echo 'SELECT column FROM table WHERE some_column = "somevalue"' | mysql -uuser -ppass --silent yourdb | awk -v q="'" '{ print q $0 q }' | paste -s -d ',' | sed 's/(.*)/[\1];/'

The first part of the command is self explanatory, you pipe in a query to mysql, and ask it to give you raw unadorned output. It will return each row for column 'column' from table 'table' as a line of output.

You pipe it to awk and ask it to wrap the values in single quotes. Due to shell escaping with single quotes, you set the q variable to a single quote. Paste then joins all the output lines together separated by commas.

Finally I use sed to wrap the resulting output in Javascript array literal '[' and ']' symbols. Awk or any other tool to concatenation approach would do just fine here too.

field_name: [Start TO Finish] 

You can also use wildcards and specific constants in a logical way e.g:

[NOW TO *] or [* TO *]

To search over documents that do not have a value for that date field, e.g. is NULL, you use the syntax:

-field_name: [* TO *].

It is hard though, to search for dates that are EITHER NULL OR lie in a specific range.

It would seem logical to specify

date_field:[Start To Finish] OR -date_field: [* TO *] 

Unfortunately Solr does not appear to support specifying a field multiple times in this way.

So the trick is to effectively query for everything you do not want, then negate the result.

This approach says select me anything that isn't in the range of this date and is not null. When you invert that result, you get all the documents sit inside the date range or are NULL.

The query to weave this magic is

-(-date_field:[Start TO Finish] AND date_field:[* TO *])

A Singleton, if you've not heard the term before, was popularised in the Design Patterns book by the Gang of Four (Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides). To be very succinct, a Singleton is a way to ensure there is only ever one instance of a class in an Object Oriented design. To put it in even simpler terms , it is an Object Oriented version of a global variable.

It's used heavily in Magento (in the app/code/core directory, 2261 times in fact!). But anyway, why is it considered harmful? There are a number of arguments for why and why not. Herb Sutter's Once is not enough gives a pretty good (and fun to read) overview of them, or you can read Kenton Varda who looks in-depth at the topic. I generally think though, that in Object Oriented software, you're seeking to create abstractions around complexity. The Singleton is a (too) convenient escape hatch from encapsulation and can lead to the attendant issues you get with global variables.

In the Magento/PHP land a more implementation specific problem with Singletons, is memory consumption. Today I was revisiting a Magento Promotions extension I had written and trying to figure out why it was suddenly obliterating PHP's memory_limit.

This extension basically piggybacks on the existing Promotions/Coupons system, but generates an index of products that match coupon codes, the price before and after the promotion is applied and some other metadata.

In order to determine what products have a promotion associated with them, I run through all the products, and match SalesRule conditions against them. I create a synthetic quote for the products that match and then pump them through the SalesRule validator. This effectively applies the promotion to the product and let us see what the savings are.

It's fairly basic, it doesn't look at multi product combinations but it works well enough for simple cases.

/**
 * Apply pricing rules to a synthetic quote to calculate discounted price
 * 
 * @param string $couponCode
 * @param Mage_Catalog_Model_Product $product
 * @return  float
 */
public function applyToProduct($couponCode, $product)
{
    $quote = Mage::getModel('sales/quote');
    $item = Mage::getModel('sales/quote_item')
        ->setQuote($quote)
        ->setProduct($product)
        ->setQty(1)
        ->setBaseDiscountCalculationPrice($product->getPrice())
        ->setDiscountCalculationPrice($product->getPrice());

    $validator = Mage::getSingleton('salesrule/validator')
        ->init(1, 1, $couponCode);
    
    $validator->process($item);
    
    return $product->getPrice() - $item->getDiscountAmount();
}

Now when I wrote this code, it seemed sensible to use the validator as a Singleton, after all I only needed one copy of it. It didn't, at the time, seem to make sense to create and then destroy the validator a couple of thousand times during indexing. Indeed when this code was first deployed, everything ran smoothly.

Recently the user of this extension added a whole bunch of sales rules - and this caused that product/salesrule index loop to detonate.

That Singleton Validator, which was written as some sort of optimization, started happily hosing over a gig of ram.

Changing getSingleton() to getModel() took ram usage down from 1100MB to about 80MB.

My suspicion is that PHPs garbage collection wasn't cleaning up adequately after each validation attempt. As the validator is effectively static, it never gives up its references for PHP to clean up. When you use getModel(), the validator loses all its references after each loop. While it means it also has to be constructed after each loop, but that allows PHP to free the memory it is using.

The Singleton is already a controversial pattern these days, but Magento developers should be particularly wary of it's implementation and its scope to hose memory.

Typically I manage PHPStorm by extracting it to /opt and then symlinking the extracted folder to /opt/PhpStorm.

To create a nice launcher for Unity you create a desktop entry under ~/.local/share/applications

$ vim ~/.local/share/applications/jetbrains-phpstorm.desktop

Now paste the following in (adjusted for your own paths)

[Desktop Entry]                                                                  
Version=4.0.1                                                                   
Type=Application                                                                
Name=JetBrains PhpStorm                                                         
Exec=/opt/PhpStorm/bin/phpstorm.sh %f                                           
Icon=/opt/PhpStorm/bin/webide.png                                               
Comment=Develop with pleasure!                                                  
Categories=Development;IDE;                                                     
Terminal=false                                                                  
StartupNotify=true                                                              
StartupWMClass=jetbrains-phpstorm           

Hit your windows or super key and type in 'Php' - you should see the newly created desktop entry there. Once launched and fully started, you can opt to keep PHPStorm locked to the Unity launcher for easy startup.

Even today, almost all x86 linux distributions can run (in theory) on a 32bit 386 processor. Let's be clear though, while it is a remarkable (and successful) processor, the 386 is an antique. The selling point of Gentoo was that it was one of the few distributions to give you the power to build a system, specifically for modern processors abandoning backwards compatibility for long-obsolete ones.

In the early days you had to start from what was called a (and I don't think they do it this way anymore) 'stage 1' install. A stage 1 install is where all you have is a bootable livecd with a few basic set of tools: a c compiler, a shell and the basic gnu coreutils...and irssi. Enough utilities to allow you to build further C packages. You'd run the command 'emerge system' and it would go off and build gcc, glibc, coreutils etc. Once these were built you'd then rebuild glibc, and gcc with your newly compiled, architecture specific, compiler and c library.

Anyway, this took a long time, particularly on sub gigahertz Pentium 2s and 3s, and Gentoo systems tended to break a lot, and by break, in the absolute best case, I mean merely became un-bootable.

The process to recover the system was pretty much the same as to install it, you had to boot from a livecd, configure your network card, hook up to the network, then chroot to the broken disk. At this point you could try and repair whatever damage you had caused.

These days if you do something silly, like I don't know, try and dist-upgrade from Ubuntu Oneiric to Precise, you can get that true Gentoo feeling (i.e. nothing works and you can't boot the machine).

This happened to me this afternoon and the hard yards done with Gentoo came to the rescue.

Here's how you do it.

Boot up from a livecd (or usbkey), get the network card modules loaded and get a dhcp address. With the Precise Live DVD you can do all of this pretty easily by selecting the 'try without installing' option from the bootloader.

Once you're online, you need to prepare the mount. First step is to mount the root partition somewhere, I typically use /mnt/ so say /mnt/oneiric (it can be whatever). If you're not sure of your partition numbers, your livecd will almost certainly come with fdisk with which you can push 'p' to print the partition table.

$ mount -t ext4 /dev/sda5 /mnt/ubuntu

If you have a separate boot partition mount that too.

$ mount -t ext2 /dev/sda1 /mnt/ubuntu/boot

Now in order to have a functional chroot, we need the proc, dev and sys subsystems to be mounted onto the chroot. This is the tricky bit.

$ mount -t proc none /mnt/ubuntu/proc
$ mount -o bind /dev /mnt/ubuntu/dev
$ mount -o bind /sys /mnt/ubuntu/sys

In the case of the sys and dev dirs, we need to reference the exact same mountpoints as the host so we use the -o bind option.

Last thing, we want to have functional network name resolution so we copy over the host's /etc/resolv.conf to /mnt/ubuntu/etc/resolv.conf

Now the chroot is ready

$ chroot /mnt/ubuntu /bin/bash
$ source /etc/profile

The chroot will be pretty much as it would be if you'd booted into it normally with a few exceptions. The kernel and kernel modules will be those of the host. If you need to access some specific hardware you need to set this up on the host.

My busted Precise install was fixed with a simple apt-get update and upgrade, before re-running the grub installer.

Forcing the cataloginventory_stock indexer to re-run fixes the situation but if you want to script the status update of many stock items, you can have a short period where your store's products will be unavailable.

Stepping through the issue I found myself in app/code/core/Mage/Catalog/Model/Resource/Product/Status.php and specifically the getProductStatusMethod()

/**
 * Retrieve Product(s) status for store
 * Return array where key is a product_id, value - status
 *
 * @param array|int $productIds
 * @param int $storeId
 * @return array
 */
public function getProductStatus($productIds, $storeId = null)
{
   $statuses = array();

   $attribute      = $this->_getProductAttribute('status');
   $attributeTable = $attribute->getBackend()->getTable();
   $adapter        = $this->_getReadAdapter();

   if (!is_array($productIds)) {
       $productIds = array($productIds);
   }

   if ($storeId === null || $storeId == Mage_Catalog_Model_Abstract::DEFAULT_STORE_ID) {
       $select = $adapter->select()
           ->from($attributeTable, array('entity_id', 'value'))
           ->where('entity_id IN (?)', $productIds)
           ->where('attribute_id = ?', $attribute->getAttributeId())
           ->where('store_id = ?', Mage_Catalog_Model_Abstract::DEFAULT_STORE_ID);

       $rows = $adapter->fetchPairs($select);
   } else {
       $valueCheckSql = $adapter->getCheckSql('t2.value_id > 0', 't2.value', 't1.value');

       $select = $adapter->select()
           ->from(
               array('t1' => $attributeTable),
               array('value' => $valueCheckSql))
           ->joinLeft(
               array('t2' => $attributeTable),
               't1.entity_id = t2.entity_id AND t1.attribute_id = t2.attribute_id AND t2.store_id = ' . (int)$storeId,
               array('t1.entity_id')
           )
           ->where('t1.store_id = ?', Mage_Core_Model_App::ADMIN_STORE_ID)
           ->where('t1.attribute_id = ?', $attribute->getAttributeId())
           ->where('t1.entity_id IN(?)', $productIds);
       $rows = $adapter->fetchPairs($select);
   }

   foreach ($productIds as $productId) {
       if (isset($rows[$productId])) {
           $statuses[$productId] = $rows[$productId];
       } else {
           $statuses[$productId] = -1;
       }
   }

   return $statuses;
}

This method goes through a list of productIds, and will assign a status id to them, this is typically used on grouped products when determining if all their children stock items are out of stock.

In testing, the status ids were all coming back as -1, i.e. not valid, and so therefore the group was out of stock.

In my code the store id was neither null, nor the default store id, so execution fell through to the else branch. At first I inserted a print_r($select->assemble()) to see the SQL being generated. The SQL was fine and when pasting it into MySQL I got a bunch of valid looking results. Funnily though, the status column was first, and the product id column was second (unlike the if branch, where they were in reverse order). This presents a problem when we reach the fetchPairs() statement.

Zend DB's fetchPairs returns an associative array resultset where column a is the key, and column b is the value. Because the SQL was returning the status column first (i.e. the key value) the result set consisted of just 2 rows (for each unique status code). In order for this code to work as you would expect the entity id (product id) needs to be first in the result set, then it gets used as a key.

The fix is straight forward enough, replace

$select = $adapter->select()
            ->from(
                array('t1' => $attributeTable),
                array('value' => $valueCheckSql))

with

$select = $adapter->select()
            ->from(
                array('t1' => $attributeTable),
                array('entity_id', 'value' => $valueCheckSql))

This way the product id is always used as the key in fetchPairs and you get a status result for each product.

• http://www.growing-object-oriented-software.com/
• http://www.mockobjects.com

The big idea is messaging. Kay98. It's a quote cited early on in Growing Object Oriented Software Guided by Tests (GOOS), a book that looks at Test Driven Development (TDD) using Mock Objects. This idea of messaging being central to Object Oriented Analysis and Design (OOAD), drives much of what is presented throughout the book.

I think as OOAD has matured over the past decade, the mode of thinking of classes as hierarchical constructs has lost favour. Increasingly OOAD is about managing the collaboration of a large number of small, independent, objects. In such designs the solution to the problem is achieved by the way the developer defines the software's object graph – i.e. its composition. In a design that is focussed on getting the composition of objects just so, communication between them is the important thing, much more so than classification.

In GOOS, TDD is presented as an exercise to first understand and then improve messaging protocols between objects. GOOS demonstrates that Mock Objects are an ideal tool to help discover these protocols . This is unusual, even in 2012, for a lot of developers. Typically I've always used Mock Objects as a Test Stub or Double. A placeholder object to induce some specific behaviour I want to test, or to isolate the unit of code I'm testing. GOOS sees Mocks a different way, as a means of representing roles within a system. The authors say a Mock Object is not a Stub, but instead an interface to some behaviour or role. Mock Roles not Objects, a paper written by Freeman, Pryce, et al. way back in 2004, explains this concept and sets out much of groundwork for GOOS.

GOOS itself is structured into five sections. The first couple of sections (very) briefly introduce the reader to TDD, the basic tenets of Agile development (very heavily influenced by Extreme Programming, XP), testing tools (JUnit, JMock, Hamcrest, Windowlicker) and the authors' OOAD philosophy.

The overwhelming impression I get from Freeman and Pryce's introduction to TDD and OOAD is that they see writing tests as less an exercise in producing a regression catching suite, and more as a design exercise. By writing a failing test up front, you have an immediate, testable, statement of intent as to what the software will do, and just as importantly, what it wont do. Focussing on just a narrow slice of behaviour, as represented by a single test, helps narrow the scope of what needs to be done. The act of satisfying the test, making it pass, focuses the developer's mind on the domain of the problem and forces both the developer and the customer to think of what from the environment impacts the test. Flushing out dependencies, object peers and services early on is a good thing.

Too often in Big Design up Front approaches, you write a ton of code according to a pre-determined (but unchecked) idea of the environment. When you go to hook everything up at the end you can find that, to the horror of your project sponsor (and spouse who wont see you for at least the next week), that, actually nothing hooks up. Or worse, you have misinterpreted what the Sponsor wanted. In a Test Driven Design, if elements of the system are incompatible, you're alerted to it early. If you have gotten the requirements completely wrong the customer can see it straight away. TDD takes 'Fail Fast' to its logical extreme.

The meat of the book is a worked example. The authors describe a fictional auction sniper system that connects to an auction, makes bids on items and either wins or loses. It's a simple example, but complicated enough to run into common issues developers face when developing OO software. Certainly there is little trouble filling out 150 pages as the authors work from an abstract set of stories to concrete code. What is good about their example is the way you see the code transforming in stages as extra features are added. The writers are careful to explain the motivations for the transformations they make and they tie it back to the TDD and OOAD principles they introduce in the first two sections.

Having worked through the Auction Sniper application using Java, JUnit, JMock, Hamcrest and Windowlicker there's a brief recap and the book moves swiftly onto it's fourth and fifth sections.

The fourth section covers 'sustainable TDD', an important and increasingly relevant topic for many developers. The burden of maintaining poorly designed test suites is a drag on developer productivity. Rather than liberating developers to improve the structure of their code, bloated indecipherable test suites become a handbrake. GOOS goes through techniques to keep test suites effective, flexible and importantly - expressive. The concept, that software is about communication, is emphasised across the book. Tests are no different. Tests should express the developers intent and function as the rough and ready documentation of a unit of code. I found the advice around constructing data builders - techniques for creating test data for use in your test cases - particularly valuable.

The final section tackles the really hard stuff, dealing with persistence (and by extension any sort of frameworky data service), asynchrony and lastly concurrency.

I found GOOS easy to read and its chapters are of a length that can be easily read on the train/bus or in short bursts. Physically, the paper is of high quality and the typesetting clear, and easy to read. I really like the images and diagrams which are simple, authentic and aid what is being discussed. I felt like the introductory sections to TDD and particularly the author's OO philosophies were fairly succinct, perhaps too succinct. But as they state from the outset, this is not a book on TDD. And anyway, Meszaros07 should be enough TDD for any human being.

GOOS comes with a particularly outstanding bibliography. Freeman and Pryce's academic backgrounds and broad reading are on display with the depth and quality of their references. As a starting point GOOS, while being a pretty domain specific (Mock Object) text, serves as a wonderful launching pad to further OOAD and TDD reading.

I like that the authors are people who practice what they preach. GOOS is a book for people who write real code in the real world. Sadly there seems too many authors, 'consultants' and coaches these days that talk a lot about programming and programming techniques yet seldom practice it in the wild. GOOS reads like a book borne out of brutal trench warfare with Objects. It is refreshing to read a book detailing a principled design philosophy and practice that has been tested in the dirty unwashed world of Enterprise

I've slowly been pruning back my physical book collection and trying to maintain a library of what I would consider 'classics'. The GOF book, Fowler's Refactoring and PoAEE Books, Beck's XP White Book, the Prag Prog Book, Kernighan and Richie's C Book, SICP and Knuth's The Art of Computer Programming series (and if I can ever get it, Mike Abrash's Black Book). I think, for me, this book fits into that category. It represents a decade of work and thinking, neatly explained by two highly skilled and above all practical developers who were at the heart of it all.

I highly recommend this book to anyone actively practising TDD and also generally to anyone with an interest in Object Oriented Software Design and Practice. While GOOS is a 'Java book' and day to day I program in PHP, the principles, practices and overarching philosophy easily translate.

Follow Steve Freeman @sf105 on Twitter, and read his blog http://www.higherorderlogic.com/. Nat Pryce is @natpryce and blogs at http://www.natpryce.com/.

Also be sure to check out the excellent mailing list http://groups.google.com/group/growing-object-oriented-software.

To map it to twig, edit vimrc and add:

au BufRead,BufNewFile *.twig set filetype=htmljinja

Generally speaking (i.e. not with Mac OSX's Terminal.app) .bash_profile gets sourced only on login. Specifically this means only when you enter your username and password from the console. The .bashrc file is sourced when starting an interactive session, that is, whenever you open up a terminal.

There is some confusion here, when you open up a login shell, such as if you use the su - command or run an explicit login shell sometimes provided by a desktop environment. In these cases the rule applies, a login shell means .bash_profile is sourced first, then .bashrc.

I tend to put environment setup in bash_profile, things like paths and any specific one-time configuration settings that aren't likely to change very much. But it's quite reasonable to just put a source .bashrc in your .bash_profile and then put everything in .bashrc.

Then, to include it in a template (say homepage.phtml):

<?php echo $this->getLayout()->createBlock('cms/block')->setBlockId('indentifer')->toHtml() ?>

  $mock = $this->getMock('Zend_Mail');
  $mock->expects($this->any())
            ->method(new PHPUnit_Framework_Constraint_IsAnything())
            ->will($this->returnSelf());

This has the effect of making any method call on your mock object return a reference to itself.

Are software patents really protecting innovation? Or are they really just protecting large businesses, increasing barriers to entry and reducing the ability for a small, innovative competitor to survive?

It's something [software patents] that's really depressing because it's so horribly wrong for someone who's a creative engineering type to think about patenting a way of thinking. Even if you had something really, really clever, the idea that you're not going to allow someone else to follow that line of thought... All of science and technology is built standing on the shoulders of the people that come before you. Did Newton patent calculus and screw the hell out of the other guy working on it? It's just so wrong, but it's what the business world does to things, and certainly the world is controlled by business interests. And technical idealists are a minority.

Electric Playground interviews John Carmack, 1997

Carmack got bitten by patents himself when he discovered a shadow drawing technique that somewhat ironically came to be known as Carmack's Reverse. While Carmack arrived at the technique independently, Creative had sought a patent for something very similar a few months earlier. The issue raised its head again this year with the Doom 3 source code release. In order to get legal approval to release the code, Carmack had to change the implementation to avoid infringing Creative's patent.

Patenting software really doesn't make sense. In programming there is usually an optimal and natural way of solving any given problem. By making optimal solutions exclusive, it reduces the efficiency with which people can create robust software. It's something that doesn't seem likely to encourage innovation to me. It seems much more likely to stifle it. And that's before we even consider the chilling effect that legal threats from gargantuan corporations can have on a would-be technologists.

Now you can manually do it yourself with apt-get purge , or, you can use the dpkg command to do it in a nice oneliner.

dpkg --get-selections | grep deinstall | sed 's/deinstall/\lpurge/' | dpkg --set-selections; dpkg -Pa

Start a VM

VBoxManage startvm "Name of VM" --type headless|gui

Forcibly Shutdown a VM

VBoxManage controlvm "Name of VM" poweroff

List VMs

VBoxManage list vms

Unregister/delete a VM

VBoxManage unregistervm "Name of VM" --delete

Forward / Unforward Ports for SSH

Modifyvm must be only be used when VM is powered off, otherwise controlvm for running VMs.

VBoxManage modifyvm "VM name" --natpf1 "guestssh,tcp,,2222,,22"
VBoxManage modifyvm "VM name" --natpf1 delete "guestssh"

$ skype
> skype: error while loading shared libraries: libXss.so.1: cannot open shared object file: No such file or directory

Hmmm.

$ locate libXss.so.1
> /usr/lib/x86_64-linux-gnu/libXss.so.1
> /usr/lib/x86_64-linux-gnu/libXss.so.1.0.0

Okay, so it's moaning about the X screensaver library, libxss, not being there. But it is there, although specifically, it's a 64bit library. I bet Skype isn't 64bit...

$ file /usr/bin/skype
> /usr/bin/skype: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, stripped

So that explains that then, we need compatible shared x86 libraries, but when you get these sorts of issues it's always best to see what else is missing. The ldd tool comes to the rescue.

$ ldd /usr/bin/skype
> linux-gate.so.1 =>  (0xf76f2000)
> libasound.so.2 => /usr/lib32/libasound.so.2 (0xf75df000)
> libXv.so.1 => /usr/lib32/libXv.so.1 (0xf75d9000)
> libXss.so.1 => not found
> librt.so.1 => /lib32/librt.so.1 (0xf75d0000)
> libQtDBus.so.4 => not found
> libQtGui.so.4 => not found
> libQtNetwork.so.4 => not found
> libQtCore.so.4 => not found
> libpthread.so.0 => /lib32/libpthread.so.0 (0xf75b4000)
> libstdc++.so.6 => /usr/lib32/libstdc++.so.6 (0xf74c9000)
> libm.so.6 => /lib32/libm.so.6 (0xf749e000)
> libgcc_s.so.1 => /usr/lib32/libgcc_s.so.1 (0xf7480000)
> libc.so.6 => /lib32/libc.so.6 (0xf7306000)
> libdl.so.2 => /lib32/libdl.so.2 (0xf7301000)
> libX11.so.6 => /usr/lib32/libX11.so.6 (0xf71cb000)
> libXext.so.6 => /usr/lib32/libXext.so.6 (0xf71b8000)
> /lib/ld-linux.so.2 (0xf76f3000)
> libxcb.so.1 => /usr/lib32/libxcb.so.1 (0xf7198000)
> libXau.so.6 => /usr/lib32/libXau.so.6 (0xf7194000)
> libXdmcp.so.6 => /usr/lib32/libXdmcp.so.6 (0xf718d000)

So looking at this, I do not have both compatible libxss and misc qt libraries installed.

With Ubuntu, to enable multiarch support, check the file /etc/dpkg/dpkg.cfg.d/multiarch. There should be a line like below (and if there isn't, add it):

# in file /etc/dpkg/dpkg.cfg.d/multiarch
foreign-architecture i386

Now we need to satisfy Skype's i386 dependencies.

 $ sudo apt-get install libxss1:i386 libqtcore4:i386 libqt4-dbus:i386 libqtgui4:i386

If we re-run ldd we can see Skype's shared library dependencies are now satisfied.

$ ldd /usr/bin/skype
> linux-gate.so.1 =>  (0xf7749000)
> libasound.so.2 => /usr/lib32/libasound.so.2 (0xf7634000)
> libXv.so.1 => /usr/lib32/libXv.so.1 (0xf762e000)
> libXss.so.1 => /usr/lib/i386-linux-gnu/libXss.so.1 (0xf7629000)
> librt.so.1 => /lib32/librt.so.1 (0xf7620000)
> libQtDBus.so.4 => /usr/lib/i386-linux-gnu/libQtDBus.so.4 (0xf75a6000)
> libQtGui.so.4 => /usr/lib/i386-linux-gnu/libQtGui.so.4 (0xf6ae0000)
> libQtNetwork.so.4 => /usr/lib/i386-linux-gnu/libQtNetwork.so.4 (0xf69a4000)
> libQtCore.so.4 => /usr/lib/i386-linux-gnu/libQtCore.so.4 (0xf6702000)
> libpthread.so.0 => /lib32/libpthread.so.0 (0xf66e7000)
> libstdc++.so.6 => /usr/lib32/libstdc++.so.6 (0xf65fc000)
> libm.so.6 => /lib32/libm.so.6 (0xf65d2000)
> libgcc_s.so.1 => /usr/lib32/libgcc_s.so.1 (0xf65b4000)
> libc.so.6 => /lib32/libc.so.6 (0xf6439000)
> libdl.so.2 => /lib32/libdl.so.2 (0xf6434000)
> libX11.so.6 => /usr/lib32/libX11.so.6 (0xf62fe000)
> libXext.so.6 => /usr/lib32/libXext.so.6 (0xf62eb000)
> libQtXml.so.4 => /usr/lib/i386-linux-gnu/libQtXml.so.4 (0xf62aa000)
> libdbus-1.so.3 => /lib32/libdbus-1.so.3 (0xf6261000)
> libfontconfig.so.1 => /usr/lib32/libfontconfig.so.1 (0xf622b000)
> libaudio.so.2 => /usr/lib32/libaudio.so.2 (0xf6211000)
> libglib-2.0.so.0 => /lib32/libglib-2.0.so.0 (0xf6118000)
> libpng12.so.0 => /lib32/libpng12.so.0 (0xf60ee000)
> libz.so.1 => /usr/lib32/libz.so.1 (0xf60d9000)
> libfreetype.so.6 => /usr/lib32/libfreetype.so.6 (0xf6041000)
> libgobject-2.0.so.0 => /usr/lib32/libgobject-2.0.so.0 (0xf5ff2000)
> libSM.so.6 => /usr/lib32/libSM.so.6 (0xf5fe9000)
> libICE.so.6 => /usr/lib32/libICE.so.6 (0xf5fcf000)
> libXi.so.6 => /usr/lib32/libXi.so.6 (0xf5fbf000)
> libXrender.so.1 => /usr/lib32/libXrender.so.1 (0xf5fb4000)
> libgthread-2.0.so.0 => /usr/lib32/libgthread-2.0.so.0 (0xf5fad000)
> /lib/ld-linux.so.2 (0xf774a000)
> libxcb.so.1 => /usr/lib32/libxcb.so.1 (0xf5f8e000)
> libexpat.so.1 => /lib32/libexpat.so.1 (0xf5f64000)
> libXt.so.6 => /usr/lib32/libXt.so.6 (0xf5f08000)
> libXau.so.6 => /usr/lib32/libXau.so.6 (0xf5f04000)
> libpcre.so.3 => /lib32/libpcre.so.3 (0xf5ec4000)
> libffi.so.6 => /usr/lib32/libffi.so.6 (0xf5ebd000)
> libuuid.so.1 => /lib32/libuuid.so.1 (0xf5eb7000)
> libXdmcp.so.6 => /usr/lib32/libXdmcp.so.6 (0xf5eb0000)

Skype now starts up and behaves as I would expect.

$ gcc -m32 -o test32 test.c

After a new install of Ubuntu 11.10 I was left scratching my head for a little bit with the following error:

> In file included from /usr/include/stdio.h:28:0,
>             from test.c:1:
> /usr/include/features.h:323:26: fatal error: bits/predefs.h: No such file or directory
> compilation terminated.

In order to build 32bit executables you need to install the i386 libc dev package.

$ sudo apt-get install libc6-dev-i386

For a little bit of 'fun' if you pass the -S -masm=intel arguments to gcc, you can compile once with -m32, and once with -m64 to see how your program changes based on target architecture. It can (perhaps it's just me :)) be Interesting to see the subtle changes in programs based on how they are assembled for x86_64 vs x86.

$ gcc -m32 -S -masm=intel test.c

Keeping up with changes to PHPUnit's suite of extensions can be occasionally dizzying, things have been put in and then ripped out of the core quite frequently in the last few releases. A common problem I'm seeing lately upgrading my test-suites to be 3.6 compatible, is the extraction of database testing classes to their own package on the pear.phpunit.de channel.

If you see an error like this:

> include(PHPUnit/Extensions/Database/TestCase.php): failed to open stream: No such file or directory

Odds are you need to do this:

$ sudo pear config-set auto_discover 1
$ sudo pear install --alldeps pear.phpunit.de/DbUnit

On a setup I administer, there are over 200,000 address records. When you view an order in the backend, and click 'edit address', the server grinds away and eventually dies either because it hits the max_execution_time limit or runs out of RAM.

You might see an otherwise meaningless error like this:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 79 bytes) in /home/somewhere/public_html/lib/Zend/Db/Statement/Pdo.php on line 290

The cause for this, is the strange manner in which Magento searches for addresses in app/code/core/Mage/Adminhtml/controllers/Sales/OrderController.

/**
 * Edit order address form
 */
public function addressAction()
{
    $addressId = $this->getRequest()->getParam('address_id');
    $address = Mage::getModel('sales/order_address')
        ->getCollection()
        ->getItemById($addressId);
    if ($address) {
        Mage::register('order_address', $address);
        $this->loadLayout();
        $this->renderLayout();
    } else {
        $this->_redirect('*/*/');
    }
}

The problem is in the $address->getCollection()->getItemById() chain. Magento creates and fully loads a collection of Address objects (which when you have 200k of them, takes a while). The final call in the chain, getItemById() takes the collection, iterates over it assigning each address to an array keyed by its entityId. It then returns any value which matches $addressId.

Now, there's another, really simple way to do the same thing. It doesn't involve instantiating 200,000 address objects, or iterating over them, or even using associative arrays. It's very familiar.

$address = Mage::getModel('sales/order_address')->load($addressId);

This one line does the same thing far more efficiently. Now, the thing that worries me, is I can't see any reason why they aren't doing this already? The change in code works, nothing appears to break and the speed boost is (obviously) immense.

So why is it not done this way?

$ varnishd -C -f /path/to/mysetup.vcl
> ...

Varnish will compile the file and print out it's full configuration as output. If there's an error, varnish handily gives you a compiler error to help trace what went wrong, which could look something like this

> Message from VCC-compiler:
> Expected an action, 'if', '{' or '}'
> ('input' Line 82 Pos 6)
>     vcl_hash(req.http.Cookie);
> -----########------------------
>
> Running VCC-compiler failed, exit 1

Some things still grate, particularly the global menus and the overlay scrollbar, but they are manageable.

A great resource I think to checkout for anyone using Oneiric is webupd8. You can find a lot of tips to bend Oneiric to your will.

For me, that means getting an official JVM installed Installing Oracle Java 7 and getting a sane desktop environment configured with global menu and overlay scrollbars zapped away Things to Tweak After Installing Oneiric.

Linux Mint (now on release 12, Lisa) is increasingly becoming a viable alternative to Ubuntu. I've played around with it a fair bit, but I still think it is not quite there yet in terms of 'just working' when compared with Ubuntu.

For someone that needs to get work done with a minimum of fuss, Ubuntu is still the leader. I do hope though, someone can work out how to marry usability and power in a slightly better way than I feel Canonical are managing at the moment.

Anyway, unlike either Enlightenment or E-Term, Imagemagick has been a firm friend ever since.

There are three main tools I find I fall back on, time and time again.

Display, Convert, and Identify.

Display, funnily enough, will open up an x-window with the contents of an image.

Convert, perhaps the most useful tool of them all, lets you resize, transform, and re-format images.

And lastly, identify, lets you get information about a file, size, format, colour depth, offset and so forth.

The power of these tools is best wielded with simple loops in bash, or using GNU Find. For example to convert a bunch of images from 1920x1080 to say 1280x720, you could do something like this

$ for IMAGE in *.jpg; do
>   # usage convert <action> orgfile.jpg newfile.jpg
>   convert -resize '1280x720' $IMAGE $(echo $IMAGE | sed 's/.jpg$/-resized.jpg/')
> done

Convert tries to be clever and maintain aspect ratios. In this case, 1920x1080 fits into 1280x720 maintaning a 16:9 aspect ratio. But what if we had a 1920x1200 input image? In these cases, convert would actually resize the image to something like 1280x800. But if we really want it to ignore common sense and squish things down to our requested 1280x720 you need to use the bang operator e.g.

$ convert -resize '1280x720!' srcimg.jpg dstimg.jpg

$ git push origin :REMOTE_BRANCH_NAME

For a concrete example of this in action

$ git branch -r
> origin/develop
> origin/master

$ git push origin :develop
> To /home/aaron/development/atestrepo.git
  - [deleted]         develop

$ git branch -r
> origin/master

So, why does this work, and why is the syntax so odd?

Well if we look at the push command in more detail it starts to make some sense.

If we want to push a local branch to a remote, we would do it like this

$ git push origin mybranch

Here, origin is the remote, and mybranch the local branch to push up. The result of this command is we now have a remote branch origin/mybranch. But what if we wanted to call the remote branch something else? We would do that like this:

$ git push origin mybranch:adiffnamefortheremotebranch

This syntax effectively translates to push to a branch adiffnameforremotebranch at remote origin the contents of mybranch. Now, can you see where this is going with respect to our delete? Deleting a remote branch with just a leading : (and no local branch name) is basically saying push nothing into remote branch called someremotebranch. Git takes this to mean delete the remote branch.

$ git push origin :someremotebranch

I find thinking of it in these terms, makes it easier to remember the syntax.

It's also worth reading the Pro Git Chapter on remotes which also covers (although, sadly, too briefly) deleting remote branches.

This is very, very handy if you want to do in-place substitutions with something like sed or in this case, the GNU text encoding conversion utility iconv.

Typically to convert between two encodings, you call iconv like this:

$ iconv -f cp1252 -t utf-8 myfile.txt

This will convert myfile.txt from windows latin 1, to utf-8 and dump the results to stdout.(Don't make the mistake of thinking iso-8859-1 and cp1252 are equivalent. It is safe to convert iso-8859-1 content as if it's cp1252, but the reverse is NOT true).

If you're in any doubt as to the encoding of the source file, you can inspect it with the 'file' command.

So if we have a directory of say c source files we want to convert, we can make use of bash, iconv and sponge to save us the tedium of converting each file manually to a new copy, then replacing the original file with the copy.

$ for FILE in *.c *.h; do iconv -f cp1252 -t utf-8 "$FILE" | sponge "$FILE"

Each file is filtered through iconv, the outputs of which are piped into sponge. Sponge soaks up the standard input until the EOF, then writes it to the original file.

A simple thing today I tried out today, downloading and extracting a web-based tarfile in a one liner.

It's really simple.

$ tar zxv < <(wget -q -O - http://www.somewhere.com/atarfile.tar.gz)
> atarfile/
> atarfile/file.txt
> ...

So let's look at this statement in a little detail. Basically we evaluate the right side of the expression first. We're using wget to download, quietly (-q), a tgz file from somewhere on the internet and output the file's contents to stdout (-O -).

We use Bash's Process Substitution operator '<(' to create a Named Pipe, which effectively creates a temporary file descriptor, and then direct the contents of that descriptor into tar using the '<' file redirection operator.

Sounds complicated but looks simple.

The UI for disabling modules is not nice, particularly once module dependencies are thrown into the mix. It's not a case of simply boring through the list of modules unchecking everything. First you need to uncheck everything you can. Save changes. Then go through the list again disabling the previously greyed out modules (because they still had active dependants) and disabling those. It can take about 4-5 passes before you're done.

Thankfully there is a much easier way using Drush. Drush is a command-line utility for managing aspects of a Drupal installation. Pertinently for Drupal upgraders it provides the ability to:

• List installed modules
• Disable modules

Running Drush without any arguments provides a list of commands and options. The two commands we are interested in are 'pm-list' (list installed modules) and 'pm-disable' (disable a module).

For these commands to work, you must issue your calls to drush from within your Drupal site's install directory e.g. /home/user/public_html/drupal

Keep in mind you can get information on any of Drush's commands by issuing a call to the help command.

$ drush help pm-list
> Show a list of available extensions (modules and themes).
> 
> Options:
>  --type                                    Filter by extension type. Choices:   
>                                            module, theme.                       
>  --status                                  Filter by extension status. Choices: 
>                                            enabled, disable and/or 'not         
>                                            installed'. You can use multiple     
>                                            comma separated values. (i.e.        
>                                            --status="disabled,not installed").  
>  --package                                 Filter by project packages. You can  
>                                            use multiple comma separated values. 
>                                            (i.e. --package="Core -              
>                                            required,Other").                    
>  --core                                    Filter out extensions that are not   
>                                            in drupal core.                      
>  --no-core                                 Filter out extensions that are       
>                                            provided by drupal core.             
>  --pipe                                    Returns a space delimited list of    
>                                            the names of the resulting           
>                                            extensions.                          
> 
> 
> Aliases: pml

We want a list of non-core modules and we would prefer non-formatted output so our command call looks like this:

$ drush pm-list --type=module --no-core --pipe
> ad
> ad_channel
> click_filter
> ad_embed
> ad_cache_file
> ad_notify
> ad_owners
> ad_report
> ...

Now can produce a list of non-core modules to pass to the disable module Drush command.

$ drush help pm-disable
> Disable one or more extensions (modules or themes). Disable dependant extensions as well.
>
> Arguments:
>  extensions                                A list of modules or themes. You can 
>                                           use the * wildcard at the end of     
>                                           extension names to disable multiple  
>                                           matches.                             
>
>
> Aliases: dis

Simple then. Putting it all together we can chain together the module list command output using shell command substitution and pass that output as arguments to the disable command, all in one neat line.

$ drush pm-disable `drush pm-list --no-core --type=module --pipe`

Press 'y' in response to the resulting confirmation prompt and the tedious work in the Drupal admin UI is replaced with a neat one-line shell command!

Branching and merging on a local repository is trivially simple, but what if you create a local branch on a repository that otherwise syncs up with a remote repository?

Follow these steps (this assumes a remote called origin is set up)

$ git branch my-new-feature
$ ... # hack away
$ git commit -a -m 'Initial feature commit'
$ git push origin my-new-feature
$ git branch --set-upstream my-new-feature origin/my-new-feature

Git pull/fetch/push will now track the newly created remote branch called 'my-new-feature'.

$ usermod -a -G wheel aaron

The -a argument is very important, it ensures arguments to -G append to the existing list of groups. Otherwise existing groups will be replaced with the argument supplied.

With an application I am currently configuring for Jenkins, it was taking about 30-35 minutes to generate all the API documentation through PHPDocumentor. I had heard good things about DocBlox - must notably its speed - and it's used by the Zend Framework so I thought I'd sub it in for PHPDocumentor and see if it was quicker.

Installation

DocBlox has a number of dependencies which we also need to install if they are not already.

• XSL PHP
• PHP-Markdown
• Graphviz

Use your system's package manager to install the xsl php extension and graphviz.

$ sudo aptitude install php5-xsl
$ sudo aptitude install graphviz

To install DocBlox and PHP-Markdown use PEAR.

$ sudo pear channel-discover pear.michelf.com
$ sudo pear channel-discover pear.docblox-project.org
$ sudo pear install --alldeps channel://pear.docblox-project.org/DocBlox-0.13.1

How Fast?

DocBlox uses largely the same command line options as phpdoc, so it can act as a drop in replacement if you already have existing build scripts utilising phpdoc.

In terms of speed, well, it's very fast. That 35 minute phpdoc run took just 43 seconds in DocBlox.

Git stash to the rescue:

$ git stash save
$ git checkout develop
$ git stash apply
$ git commit -m 'Apply stashed changes'

To start developing on this branch instead of the master, use the following command

$ git checkout -b develop origin/develop
> Branch develop set up to track remote branch develop from origin.
> Switched to a new branch 'develop'

This command creates a local branch tracking the remote develop branch and switches your working directory to this branch.

If you already have a local develop branch and want it to track the remote you can use this instead

$ git checkout develop
$ git branch --set-upstream develop origin/develop

$ sudo pear channel-discover pear.zfcampus.org
$ sudo pear install zfcampus/ZF

When configuring applications, if you wish to use an application's bundled copy of ZF, ensure local libs come first in PHP's include_path.

To solve the issue change into your jenkins home dir (/var/lib/jenkins on Ubuntu) and add the following to a file named .gitconfig.

[user]
    name = Jenkins
    email = jenkins@localhost

That will allow you to avoid this:

Started by user anonymous Checkout:workspace / /var/lib/jenkins/jobs/Bookings/workspace - hudson.remoting.LocalChannel@289d9155 Using strategy: Default Checkout:workspace / /var/lib/jenkins/jobs/Bookings/workspace - hudson.remoting.LocalChannel@289d9155 Cloning the remote Git repository Cloning repository origin Fetching upstream changes from file:///home/aaron/Sites/Bookings Seen branch in repository origin/HEAD Seen branch in repository origin/master Commencing build of Revision fc8a70bc628edb0ac547b3fefc0841db5e06204c (origin/HEAD, origin/master) Checking out Revision fc8a70bc628edb0ac547b3fefc0841db5e06204c (origin/HEAD, origin/master) FATAL: Could not apply tag jenkins-Bookings-1 hudson.plugins.git.GitException: Could not apply tag jenkins-Bookings-1 at hudson.plugins.git.GitAPI.tag(GitAPI.java:698) at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1181) at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1129) at hudson.FilePath.act(FilePath.java:758) at hudson.FilePath.act(FilePath.java:740) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1129) at hudson.model.AbstractProject.checkout(AbstractProject.java:1193) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:555) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:443) at hudson.model.Run.run(Run.java:1376) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:175) Caused by: hudson.plugins.git.GitException: Error performing command: git tag -a -f -m Jenkins Build #1 jenkins-Bookings-1 Command "git tag -a -f -m Jenkins Build #1 jenkins-Bookings-1" returned status code 128: *** Please tell me who you are.

Run

git config --global user.email "you@example.com" git config --global user.name "Your Name"

to set your account's default identity. Omit --global to set the identity only in this repository.

fatal: empty ident <jenkins@blizzard.(none)> not allowed

at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:744)
at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:709)
at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:719)
at hudson.plugins.git.GitAPI.tag(GitAPI.java:696)
... 12 more

Caused by: hudson.plugins.git.GitException: Command "git tag -a -f -m Jenkins Build #1 jenkins-Bookings-1" returned status code 128: *** Please tell me who you are.

If you do not have Jenkins installed or are not sure you have your environment setup correctly, please checkout my previous tutorial Getting Started With Jenkins for PHP.

A quick checklist

Before we start, make sure you have your environment and Jenkins installation up and running and appropriately configured.

• You have a JVM and Jenkins installed running at http://localhost:8080
• You have the following Jenkins plugins installed
  • checkstyle
  • cloverphp
  • dry
  • htmlpublisher
  • jdepend
  • plot
  • pmd
  • violations
  • xunit
• You have the following PEAR packages installed
  • pdepend/PHP_Depend
  • phpmd/PHP_PMD
  • phpunit/phpcpd
  • phpunit/phploc
  • PHPDocumentor
  • PHP_CodeSniffer
  • phpunit/PHP_CodeBrowser
  • phpunit/PHPUnit
  • phpunit/ppw
• You have Ant installed
• You have Git installed and are familiar with its basic use
• You have the Zend_Framework installed globally as a PEAR package

If you don't have the above installed please refer to the Getting Started With Jenkins for PHP tutorial.

If you're not familar with Git, have a quick read of git-scm.org or refer to my Git cheatsheet.

To install the Zend Framework as a PEAR package see my post on Installing Zend Framework from PEAR.

Create your project

I'm basing this tutorial on a sample Zend Freamework application, you're free to use whatever codebase you like. But for the purposes of this tutorial I'm using this project as a starting point: git@github.com:ajbonner/Bookings.git.

$ git clone https://github.com/ajbonner/Bookings.git ./bookings
$ cd bookings
$ ppw --name 'Bookings' --source ./application .

This will generate an ant build file with some general defaults for PHP_CodeSniffer and PHP Mess Detector along with a standard phpunit configuration file.

First thing, we'll need to make a small amendment to the phpunit configuration file, since we're testing a Zend Framework application, we need to make sure PHPUnit loads a custom bootstrap before running tests. Open up the generated phpunit.xml.dist file and change the opening phpunit element to look like this:

<phpunit backupGlobals="false"
              backupStaticAttributes="false"
              strict="true"
              verbose="true"
              colors="true"
              bootstrap="tests/bootstrap.php">

Our build environment is now configured correctly to run Zend Framework Test Cases, we have our ant build file setup and some defaults configured for our code coverage and analysis tools. Let's add our new build and phpunit configuration files to git and commit our changes. Then we can run ant and see what happens.

$ git add build.xml phpunit.xml.dist
$ git commit -m 'Add build and test configuration files'
$ ant
>> Buildfile: /home/aaron/Sites/Bookings/build.xml
>> clean:
>>    [delete] Deleting directory /home/aaron/Sites/Bookings/build/api
>>    [delete] Deleting directory /home/aaron/Sites/Bookings/build/code-browser
>>    [delete] Deleting directory /home/aaron/Sites/Bookings/build/coverage
>>    [delete] Deleting directory /home/aaron/Sites/Bookings/build/logs
>>    [delete] Deleting directory /home/aaron/Sites/Bookings/build/pdepend
>>     [mkdir] Created dir: /home/aaron/Sites/Bookings/build/api
>>     [mkdir] Created dir: /home/aaron/Sites/Bookings/build/code-browser
>>     [mkdir] Created dir: /home/aaron/Sites/Bookings/build/coverage
>>     [mkdir] Created dir: /home/aaron/Sites/Bookings/build/logs
>>     [mkdir] Created dir: /home/aaron/Sites/Bookings/build/pdepend
>> 
>> parallelTasks:
>> 
>> phpcpd:
>> 
>> pdepend:
>>      [exec] PHP_Depend 0.10.5 by Manuel Pichler
>>      [exec] 
>>      [exec] Parsing source files:
>>      [exec] ....                                                             4
>>      [exec] 
>>      [exec] Executing Coupling-Analyzer:
>>      [exec]                                                                 18
>>      [exec] 
>>      [exec] Executing CyclomaticComplexity-Analyzer:
>>      [exec] .                                                               24
>>      [exec] 
>>      [exec] Executing Dependency-Analyzer:
>>      [exec]                                                                 17
>>      [exec] 
>>      [exec] Executing Inheritance-Analyzer:
>>      [exec]                                                                  5
>>      [exec] 
>>      [exec] Executing NodeCount-Analyzer:
>>      [exec]                                                                 14
>>      [exec] 
>>      [exec] Executing NodeLoc-Analyzer:
>>      [exec] phpcpd 1.3.2 by Sebastian Bergmann.
>>      [exec] 
>>      [exec] 0.00% duplicated lines out of 119 total lines of code.
>>      [exec] 
>>      [exec] Time: 0 seconds, Memory: 2.75Mb
>>      [exec] .                                                               22
>>      [exec] 
>>      [exec] Generating pdepend log files, this may take a moment.
>>      [exec] 
>>      [exec] Time: 00:00; Memory: 9.50Mb
>> 
>> phpcs:
>> 
>> phpmd:
>>      [exec] Result: 1
>> 
>> phploc:
>>      [exec] phploc 1.6.1 by Sebastian Bergmann.
>>      [exec] 
>>      [exec] Directories:                                          1
>>      [exec] Files:                                                4
>>      [exec] 
>>      [exec] Lines of Code (LOC):                                119
>>      [exec]   Cyclomatic Complexity / Lines of Code:           0.08
>>      [exec] Comment Lines of Code (CLOC):                        18
>>      [exec] Non-Comment Lines of Code (NCLOC):                  101
>>      [exec] 
>>      [exec] Namespaces:                                           0
>>      [exec] Interfaces:                                           0
>>      [exec] Classes:                                              4
>>      [exec]   Abstract:                                           0 (0.00%)
>>      [exec]   Concrete:                                           4 (100.00%)
>>      [exec]   Average Class Length (NCLOC):                      22
>>      [exec] Methods:                                              9
>>      [exec]   Scope:
>>      [exec]     Non-Static:                                       9 (100.00%)
>>      [exec]     Static:                                           0 (0.00%)
>>      [exec]   Visibility:
>>      [exec]     Public:                                           8 (88.89%)
>>      [exec]     Non-Public:                                       1 (11.11%)
>>      [exec]   Average Method Length (NCLOC):                     10
>>      [exec]   Cyclomatic Complexity / Number of Methods:       1.89
>>      [exec] 
>>      [exec] Anonymous Functions:                                  0
>>      [exec] Functions:                                            0
>>      [exec] 
>>      [exec] Constants:                                            0
>>      [exec]   Global constants:                                   0
>>      [exec]   Class constants:                                    0
>> 
>> phpunit:
>>      [exec] PHPUnit 3.5.14 by Sebastian Bergmann.
>>      [exec] 
>>      [exec] ...
>>      [exec] .
>>      [exec] 
>>      [exec] Time: 1 second, Memory: 19.25Mb
>>      [exec] 
>>             OK (4 tests, 16 assertions)
>> 
>>      [exec] Writing code coverage data to XML file, this may take a moment.
>>      [exec] 
>>      [exec] Generating code coverage report, this may take a moment.
>> 
>> phpcb:
>> 
>> build:
>> 
>> BUILD SUCCESSFUL
>> Total time: 3 seconds
>> 

If you have everything setup correctly you will have a build directory with a number of artifacts: code coverage analysis, code style reports, dependency analysis, etc. You can look at these directly but it's much more convienient to use Jenkins to view these artifacts.

Importing a Project into Jenkins

We have a working build and we are producing a number of artifacts representing certain aspects of our software. Time now to import the project into Jenkins and have this process ran automatically.

Step 1 - Create a New Jenkins Project

As part of getting our environment ready for Jenkins, we installed a PHP Project Template. To create our new Project, we copy this template, and give our project a name. Choose whatever you like, but for the purposes of this tutorial, I've chosen 'Bookings'.

Step 2 - Configure your New Project

You're now presented with a (at first glance) daunting configuration page. We'll work our way down the page, for the most part we don't need to make major changes.

Firstly the pdepend task generates a pair of SVG images, in order to have these display on the project dashboard we need to substitute 'job-name' to 'Bookings' (or whatever you chose for your project name) in the embed tags in the description field.

Next, uncheck 'disable build', naturally we want to perform builds!

The project will be retrieved from the git repository we created during the initial build configuration. Pass in the absolute filesystem path to your repository, for example mine is file:///home/aaron/Sites/Bookings.

That's it. You now have setup the project in Jenkins and defined a source. The remaining details can be left at their defaults. Scroll down to the bottom of the page and click the save button.

Step 3 - Build the Project

The project needs to first do a build before the project workspace can be initialised. On the left hand project menu is a 'build now' link. Click that and Jenkins will checkout a copy of our project from git then process the ant build.xml we defined.

Once the build is complete you can click on the build and take in all the dianostic and code analysis artifacts generated during the build.

Possibly, you may run into an issue building your project from git. A common cause is Jenkins being unable to find a user identity for its operating system user. Please see my post Cannot Build Git Project in Jenkins if this happens to you.

What Next?

Continuous integration isn't about pretty graphs and pithy build summaries. It's about maintaining working software and highlighting issues as soon as possible. Jenkins has a wealth of plugins that help this process, but perhaps the first one to checkout is the email functionality, letting your development team know immediately if something is broken.

Assuming you're running your server on port 8080, obtain the cli jar file like this:

$ wget http://localhost:8080/jnlpJars/jenkins-cli.jar

To see a list of available commands simply run the tool with the help argument:

$ java -jar jenkins-cli.jar -s http://localhost:8080 help
> build
>   Builds a job, and optionally waits until its completion.
> clear-queue
>   Clears the build queue
> connect-node
>   Reconnect to a node
> copy-job
>   Copies a job
> ...

In recent versions of Jenkins, typical server operations have been decoupled from the cli tool and are now issued using simple http requests. For example to reload Jenkins instance's configuration, you would just fire a http request at it like this:

$ curl http://localhost:8080/reload

There are three commands of this sort:

• reload Reload server configuration
• restart Restart the server
• exit Close the server down

Issue these in the format:

$ curl http://[jenkins-server]/[command]

There's two options, you can download a pre-prepped specific package and install using aptitude or synaptic, or you can set-up an update site to ensure you're always notified of updates.

I prefer the latter and used the following steps:

$ wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
$ sudo echo 'deb http://pkg.jenkins-ci.org/debian binary/' >> /etc/apt/sources.list
$ sudo apt-get update
$ sudo apt-get install jenkins

By default, the Jenkins packages attempt to start up a java web container running on port 8080. To change this, open up /etc/default/jenkins and look for the following:

# port for HTTP connector (default 8080; disable with -1)
HTTP_PORT=8080

Change the port number to whatever works for you and (re)start the service

$ sudo service jenkins restart

magento-quickies:

When you’re creating custom Magento models, it’s common to to add and remove fields to and from a database table during development before finalizing a set of columns for the setup resource/migration file. One thing to keep in mind while you’re doing this is that Magento will cache the list of table columns each model knows about, which means a cache clearing will be needed after adding a column.

Another great thing it can do, is resume broken file transfers e.g. from a sftp, scp, http or ftp transfer.

rsync --partial --progress --rsh=ssh localfile remoteuser@remotehost.com:/remote/path

There is some trace though, a hs_* dump file in your home directory. In that file is a message (along with a lot of other verbiage) like this

# Problematic frame:
# C [libswt-pi-gtk-3659.so+0x3c17e]
Java_org_eclipse_swt_internal_gtk_OS_GTK_1ACCEL_1LABEL_1GET_1ACCEL_1STRING+0x0

Not really helpful but it helped google lead me to a solution. That solution is here: https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/789123 (see comment 18).

To summarise, you basically need to uninstall overlay-scrollbar and liboverlay-scrollbar-0.1-0.

This can be achieved either using the package manager or aptitude

sudo aptitude remove overlay-scrollbar liboverlay-scrollbar-0.1-0

This has the added benefit of removing the annoying scrollbars that come with Natty.

Turns out with a little

gem install image_size

It's pretty easy!

Personally, I maintain a fairly vanilla setup. I use the PHP Developer Tools, E/JGIt plugins and Subclipse for SVN.

To get hold of these packages add the following two update sites after installing indigo.

• http://download.eclipse.org/releases/indigo - Egit and PDT 3
• http://subclipse.tigris.org/update_1.6.x - Subclipse SVN (much better than Subversive)

Turns out there's already something there to do it

Mage_Payment_Method_Abstract::_debug($data);

If you want to call it from outside the payment_method inheritance tree use

Mage_Payment_Method_Abstract::debugData($data);

In both cases your payment method needs to have its debug config setting enabled e.g. for my usaepay module

echo Mage::getStoreConfig('payment/usaepay/debug'); 
>> 1

This is where the git remote add command comes in handy.

$ git remote add origin git@github.com:ajbonner/foo.git

You can see the remotes associated with your branch:

$ git remote -v
>> origin	git@github.com:ajbonner/foo.git (fetch)
>> origin	git@github.com:ajbonner/foo.git (push)

To push the existing code to the remote:

$ git push origin master

You now have set up the remote and pushed your master branch into it. From here it gets tricky because subsequent git pull requests will give you an ugly error

$ git pull
>> You asked me to pull without telling me which branch you
want to merge with, and 'branch.master.merge' in
your configuration file does not tell me, either. Please
specify which branch you want to use on the command line and
try again (e.g. 'git pull <repository> <refspec>').
See git-pull(1) for details.

If you often merge with the same branch, you may want to
use something like the following in your configuration file:

[branch "master"]
remote = <nickname>
merge = <remote-ref>

[remote "<nickname>"]
url = <url>
fetch = <refspec>

See git-config(1) for details.

There a few ways to get around this:

1. Follow the directions given by git and edit your gitconfig

2. Clone a fresh copy of the master branch from the remote

3. When defining the remote add the --track option and give it the name of the master branch

   $ git remote add --track master origin git@github.com:ajbonner/foo.git

4 Refer to the remote branch using --set-upstream

$ git branch --set-upstream master origin/master

Personally I find option number 4 the best with the least amount of work.

You can also use:

$ git config --global branch.autosetupmerge true

To avoid having to do this.

On my most recent project I exclusively employed Mage_Test for my Unit Testing and with a few exceptions, it's performed extremely well.

Mage_Test takes a very hands-off, light-weight approach. EcomDev meanwhile, on the surface, appears to provide far greater support for testing at the expense of some complexity.

Installing EcomDev's Test module is easy. You can install it from Magento Connect or you can get the module directly from the developer's Subversion repository

$ svn co http://svn.ecomdev.org/svn/ecomdev-phpunit/trunk

Once you have the code, it's just a matter of copying into your magento store as with any other module:

$ cp -r <ecomdevtestdir>/* path/to/your/magento/rootdir

Once the files are in place, you'll need to edit app/etc/local.xml.phpunit and supply some details for a test database connection and some path information relating to the store site root URIs.

Once your database and paths have been defined initialise the test database by changing into your store root and running the UnitTests.php test suite

~/Sites/my/store/root: $ phpunit UnitTests.php

This will take some time (one of my clients has a 400meg database and a lot of orders, it took ~ 4 minutes).

You're now ready to start writing some tests.

For more information on EcomDev's PHPUnit module see this blog post or get a copy of the (very) comprehensive EcomDev PHPUnit 0.20 Manual

mysqldump -uuser -p mydb mytables_*

There isn't an inbuilt mechanism to do this, my reply was to use a shell script with an array containing a list of tables you wanted to export. Another reply had a better way, using a call to mysql to get a list of tables matching a glob pattern, putting them in an array and then iterating over that array with successive calls to mysqldump.

I put this suggestion together with mine, made it more generic and popped it on github for reference.

Personally I've placed this in my user's .bash_functions (included by .bashrc).

There is an issue at the moment when you try and install the egit/jgit plugins with Indigo versions RC3 and above.

You'll encounter an error similar to this:

An error occurred while collecting items to be installed session context was:(profile=epp.package.rcp, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect, operand=, action=). No repository found containing: osgi.bundle,org.eclipse.egit,1.0.0.201106011211-rc3 No repository found containing: osgi.bundle,org.eclipse.egit.core,1.0.0.201106011211-rc3 No repository found containing: osgi.bundle,org.eclipse.egit.doc,1.0.0.201106011211-rc3 No repository found containing: osgi.bundle,org.eclipse.egit.ui,1.0.0.201106011211-rc3 No repository found containing: osgi.bundle,org.eclipse.jgit,1.0.0.201106011211-rc3

This is caused by the latest jgit packages not yet being in the indigo release p2 update repository. To get these installed until these updates are provided, add http://download.eclipse.org/egit/staging/ as an update site and you'll be able to install these plugins successfully.

I found this solution here: https://bugs.eclipse.org/bugs/show_bug.cgi?id=348183

$keys = array('mykey', 'another_key');
$array = array();
foreach ($keys AS $key) { $array[$key] = "hello world\n"; }
foreach ($keys AS $key) { echo $array[$key]; }

>> hello world
hello world

In Ruby, the presence (and use of Symbols) makes this a bit tricky:

keys = ['mykey', 'another_key']
myarray = { :mykey => 'hello world', 'another_key' => 'goodbye world' }
myarray.each { |k| 
  puts myarray[k]
}
 
>> hello world
goodbye world

Now if we try this again using instead the Strings from the keys array we get a different result:

keys.each { |k| 
  puts myarray[k]
}
 
>>
goodbye world

We don't get the first array value back because a String key is not the same as a Symbol key, even if they consist of the same sequence of characters. This is one of the gotchyas with Ruby.

In Ruby the 'value of a Symbol is not the same as that of a String. So :key != "key". From an ease of use perspective it would be convenient if they did.

Thankfully the language stewards saw fit to include String.to_sym as a convenience method to create a Symbol from a String's value.

keys.each { |k| 
  puts myarray[k.to_sym]
}
 
>> hello world

Pull down the 'classic' version Eclipse 3.7 Indigo from http://www.eclipse.org/downloads/ and install.

Once installed, launch Eclipse and navigate to Help->Install new Software.

Add the Indigo update site 'http://download.eclipse.org/releases/indigo'. This will take sometime to add, let it go for 5 or so minutes.

UPDATE 20/10/2011 The following step is no longer required as the the PDT 3.0 series is now in the main indigo update repository. Once the Indigo Update Site is added, add the PDT 3.0 Update Site http://download.eclipse.org/tools/pdt/updates/3.0/milestones/

Now, to install simply select PDT Development Tools All in One SDK (leave the others unselected) and click next. The installation process shouldn't take more than a few minutes. .

I had a feeling I should give them respect as much of the learning Ruby literature says to basically ignore them for now, or launch into lengthy defences of why they exist (usually forgetting to explain how they work).

In case you're unfamiliar with Ruby syntax, a Symbol looks like this

:a_symbol

With the numerous ways of defining variables in Ruby this is another variable looking constuct, except that it isn't actually a variable at all (you cannot assign to it).

So what is a Symbol, actually. To be honest I'm still struggling for a good simple definition. I think the best way to think of it, is as a constant (i.e. immutable) placeholder for the name of something. Or more practically put, a colon in front of the string you want to use for a hashkey. Agile Programming With Rails 4th Ed puts it simply 'a Symbol is a name for something'.

I wrote on Twitter that I felt Symbols were extra syntax put in to solve a problem a programmer in a high level interpreted language shouldn't have to worry about. Memory management. In Ruby everything is an object, and objects are much bigger than the primitives you have in Java, C or even PHP. A string for example in PHP (and as in C) is still ultimately a sequence of bytes stored in contiguous memory addresses. In Ruby, a string is always an object, and that requires a fair chunk of memory to represent. If you have a bunch of hash tables, using objects as a hashkey is inefficient and wasteful.

Symbols help solve this inefficiency. A Symbol is still an object, but a special one. It has few methods the main ones being to get its string value and another its integer value, it's immutable and there's only ever one copy of it. So it's much more efficient to use a symbol as a hashkey than a Ruby string. In Ruby (and most OO languges) two strings even if consisting of the same sequence of characters are different objects. In Ruby two symbols of the same sequence of characters are the same object. In large applications this feature can save a tremendous amout of memory.

The other key characteristic of Symbols is immutability. Symbols cannot be assigned to, they just are. In a language that lacks a true constant construct (uppercasing a variable name is the convention for defining a Ruby constant but read-only access isn't enforced at runtime, you can write to what Ruby calls a 'constant'), Symbols can be useful.

So are Symbols a good language feature? At this point with my lack of experience with Ruby I don't really feel qualified to answer that yet definitively. My gut though, says no. I feel in a high level language the need for extra syntax to optimise code adds an unnecessary burden on the programmer. They seem so out of place in a language that works so hard to strip away unneccessary syntax. Symbols, to me detract from Ruby's power to define elegant and natural sounding expressions.

The difficulty authors have in describing what Symbols are, how to use them and why they should be used seems like a language design smell to me. I dare say as I become more familiar and accustomed to the presence and use of Symbols I'll learn to accept them. But as a Developer new to Ruby, they seem out of place.

I haven't gone into an excessive amount of detail about the nature and application of Symbols. For that the two best resources I've found explaining are http://glu.ttono.us/articles/2005/08/19/understanding-ruby-symbols and http://www.troubleshooters.com/codecorn/ruby/symbols.htm.

The Basics

The process for creating and restoring a snapshot is trivial

$ mysqldump -uuser -p mydatabase > mydump.sql    # export
$ mysql -uuser -p mydatabase < mydump.sql        # import

For the most part this works as you expect it to, and for small databases this is probably all that is needed. You may find the resultant .sql file is huge, it is after all uncompressed text, at 1 byte per character if the output is ANSI or up to 4 bytes if your character set is UTF-8. Bzip2 will bring the file size down a considerable amount but it’s also considerably slower than gzip.

It’s sometimes tempting to gzip/bzip2 your datadump while performing the mysqldump in a single line.

$ mysqldump -uuser -p mydatabase | bzip2 -c > mydump.sql.bz2

While it seems a nice efficient way to do your backup, this should be avoided as (by default in MyISAM) you’re locking tables and denying other clients access to them. InnoDB implements row level locking which is slightly less offensive, but still should be avoided as much as possible.

When importing a large database, the choice of zip format is important. You have to trade off decompression speed, with filesize. The extra cputime consumed decompressing a bzip2 datadump may actually be less preferable to a few extra megabytes gained by using the faster gzip. Whatever your choice, importing a zipped datadump is very easy.

$ gunzip -c mydump.sql.gz | mysql -uuuser -p mydatabase  # importing a gzipped datadump
$ bunzip2 -c mydump.sql.bz2 | mysql -uuuser -p mydatabase # importing a bzip2 datadump

Locking and Transactional Integrity

As explained briefly above, MySQL’s storage engines come with some limitations. In the worst case, with MyISAM, while performing a mysqldump entire tables will be locked, that means other clients will not be permitted to write to a table while the dump is being performed. If you have large MyISAM tables this poses clear problems when backing up a running application.

InnoDB is slightly better because it uses row level locking. It locks only the rows affected by a query, and not the whole table. This makes a conflict far less likely to occur while performing a backup on a running application. InnoDB as a transactional storage engine does allow for the possibility that active transactions may be underway while you’re attempting your backup.

Two options we can pass to mysqldump mitigate these issues are, —single-transaction and —skip-lock-tables.

$ mysqldump --single-transaction --skip-lock-tables mysql -uuser -p mydatabase

The use of —single-transaction means mysql issues a begin statement before dumping the contents of a table, ensuring a consistent state of the table without blocking other applications. It means writes can occur while the backup is taking place and this will not affect the backup. The —skip-lock-tables option stops MyISAM tables being locked during the backup. This does mean the integrity of the table can be lost as writes occur to them during the backup process. The risk is weighed up against the risk of blocking access to the table during a lengthy backup process.

Improving Import Performance

Choice of zip format will have a large bearing on import performance. Gzip is appreciably faster than bzip2. Other options you can pass to mysqldump to improve import performance are —disable-keys and —no-autocommit.

Disabling keys significantly improves the performance of imports as mysql will only index the table at the end of the import. With keys enabled, the index is updated after each row is inserted. Given you are performing a batch import, this is suboptimal.

By default each statement in an InnoDB table is autocommitted. This comes with unneccessary overhead when performing a batch import as you really only need to commit once the table has been fully imported.

Further Reading

This only a brief look at using Mysqldump for backups. It’s a common enough development task that all developers should take the time to see how it can be best leveraged for their environment. There’s plenty of documentation out there on using the tool. But the best place to start is with the official docs.

"The quick brown fox", "The quicker brown fox".

If you try and use the VIM substitute regexp s/"The quick brown.*"// you'll end up nuking the whole line up to the terminating period mark. This is because the regular expression is acting greedily, and matching up to the second and final " character.

In PERL compatible regexps you would simply modify the expanding part of the regexp to be .*? which makes the expression match as little as possible.

In VIM the syntax is slightly different. To do a non-greedy expansion, you use .\{-}. For example:

s/"The quick brown.\{-}"//

This will leave you with 

, "The quicker brown fox".

$ git init <dirname>

Creates a new bare git repository in the current directory, or if a directory argument is given, in that directory.

Version control a file

$ git add file|dir <file|dir> ...

Adds, or to use git terminology ‘stages’ a file for a local commit.

Remove a version controlled file

$ git rm -rf --cached <file|dir>

This removes a file that has been previously staged (i.e added) for local commit. It does not remove the local file. If you omit the —cached option, the local file WILL be deleted. The rm command is roughly analagous to: $ svn del

Make a local commit

$ git commit <-a> -m 'Commit message' <file> ...

The -a flag will commit all pending files recursively in the current path, alternatively you can specify the exact files to commit at the end of the command separated by spaces.

See pending actions or status

$ git status <-u no|all|normal> <path> ...

Will show pending files and untracked files in the current working path. Optionally the -u (—untracked-files) option can be supplied to hide/show untracked files in the status report. Passing a path, or number of paths will run the status for that path rather than the current directory.

Create a bare centralised remote repository

$ git clone --bare pathtoinitialrepo <myrepo.git>

Alternatively if you don’t have an existing repo already:

$ git init --bare <path>

Often you’ll see the .git extension bandied about, particularly on github. This is a canonical reference to a bare git repository (i.e. only has the meta information and not a working copy) that is usually acts as a master or central repository.

Configuring a local repository to push to a remote

$ cd pathtoinitialrepo
$ git remote add origin ssh://myuser@somehost.com/~/repos/myrepo.git
$ git push origin master

If you initially clone from the remote repository you can skip the first step of setting up the origin. However if you have just created a new bare remote repository and want to setup an initial commit, you need to setup the origin first. Subsequent pushes to the remote repository can be done with git push origin master. This is the familiar model of subversion and svn commit.

Create a local copy of a remote git repository

$ git clone ssh://myuser@somehost.com/~/repos/myrepo.git

Create a local copy from remote repository created following the above approach.

Ignore changes to a tracked file

$ git update-index --assume-unchanged <file>

Use this command if you’ve added a file, e.g. a database defaults config file, but do not want further changes to be picked up for it.

To resume tracking changes to the file use:

$ git update-index --no-assume-unchanged <file>

I'm still very new to Ruby and its tooling so I was finding it hard to see the details of available remote gems.

I was labouring:

$ gem search --remote tumblr

*** REMOTE GEMS ***
integrity-tumblr (0.1.1)
matenia-tumblr-api (0.1.6)
ruby-tumblr (0.0.2)
skinnycms_tumblr (0.0.2)
tumblr (2.2.0)
tumblr-api (0.1.4)
tumblr-rb (1.3.0)
tumblr4r (0.7.2)
tumblr_cleanr (0.0.1)

Then for each interesting result: 

$ gem specification --remote <gem>

This brought back a lof interesting information but it is hardly an efficient way to browse. Digging into the docs I found an easier way.

$ gem search -r -d tumblr

*** REMOTE GEMS ***
integrity-tumblr (0.1.1)
    Author: Matías Flores
    Rubyforge: http://rubyforge.org/projects/integrity
    Homepage: http://github.com/matflores/integrity-tumblr
    Tumblr notifier for the Integrity continuous integration server
 
matenia-tumblr-api (0.1.6)
    Author: Jeff Kreeftmeijer, Matenia Rossides
    Homepage: http://github.com/matenia/tumblr
    Tumblr API wrapper - maintained by matenia

The -r is the short notation for --remote, and -d is the short notation for --details. The default for the search action is --no-details, which is why the initial resultset was so unhelpful.

$ sudo pear upgrade-all
> PHP Fatal error:  Call to undefined method PEAR::raiseErro() in /usr/share/php/PEAR/REST.php on line 165
PHP Stack trace:
...

You can fix this nuisance:

$ sudo mkdir -p /tmp/pear/cache
$ sudo pear upgrade-all

Hope this helps save someone some time!

$ gem server

Open your browser and go to 127.0.0.1:8808 and read all the api docs for your installed gems (i.e Rails and friends).

That, is very cool!

Typically a PHP project will start off small, some basic webpage views, maybe a few forms, and likely, some sort of search functionality. This is pretty basic and if things need to change, you can normally change it in place, directly on the web-server and without too much grief.

At some point though entropy takes its toll and incremental changes have so may unintended side-effects it's no longer feasible to safely make edits in-place. A quick hack directly on the web-server stops being a 'quick win', and more like a game of Russian Roulette.

This is where unit and integration testing comes to the fore. A safety net to protect us when we start changing an application. But often getting a test environment set-up and representative of the live system is a lot of work in itself, and the temptation to just give into the 'Inner Pig' is too great. That is, to not bother running any tests, and just cross fingers.

Jenkins makes this a lot easier by doing much of the heavy lifting of building and running tests. It will check that code complies with defined style conventions. It can also check for common coding smells (copy/paste, duplication, long methods, large classes, poor expressions) and it can run custom scripts depending on the success of unit and integration tests.

Essentially Jenkins ensures that any changes that go into source control, do not 'break the build' in a process known as Continuous Integration.

Installation is very easy. Visit the Jenkin's site, follow the instructions for your platform and you'll have a Jenkins's install running (by default) at http://localhost:8080. I have also written a small installation tutorial: Installing Jenkins on Ubuntu/Debian Systems.

To get Jenkins initially dressed up for PHP follow the jenkins-php.org site upto and including the setup the Pear packages and Jenkins plugins.

You may or may not run into trouble with these instructions. For the most part it worked fine for me, however I chose to use the cli-tool and the plugin repository was not initialised. To get up and running, I had to force the plugin list to refresh manually.

Manage Jenkins > Manage Plugins > Click the Advanced Tab

If you're like me and want git or svn scm access, you'll want to install these plugins as well, as they are not included in the list on the jenkins-php instruction page.

Once the PEAR packages and Jenkins Plugins are installed, you're now ready to start preparing your application for Continuous Integration.

The initial configuration can be quite terse, as you will need an initial ant build file, and sample configurations for PHP Code Sniffer and the other code analysis tools. Thankfully Sebastian Bergmann - author of PHPUnit and much of the Jenkins PHP suite of tools - has developed a project wizard utility to simplify these initial configuration steps.

Install the PHP Project Wizard from the PHPUnit channel:

$ sudo pear install phpunit/ppw

Once installed, you can change into your project dir and run it with a few arguments to setup your initial build environment.

$ ppw --name 'My Project' --source ./lib --tests ./tests

You can also specify arguments defining default rulesets for PHP Code Sniffer and PHP Mess Detector. Omitting these arguments sees ppw select some sane defaults for you.

Now install the Jenkins PHP Job Template:

$ cd path/to/jenkins/jobs
$ git clone git://github.com/sebastianbergmann/php-jenkins-template.git php-template
$ chown -R jenkins:nogroup php-template/
$ curl http://localhost:8080/reload # set this to be the path:port to your jenkins server

Jenkins and your system are now ready to manage a PHP Project. Please see my tutorial on Setting up a PHP Project in Jenkins for how to setup your first project in Jenkins.

It's similar with programming languages, some familiar constructs will work exactly the same as you expect. Some wont work at all, and in some cases, they'll sort of work. 

In Ruby, if you're like me and come from a C, Unix or PHP background, one of the first things to trip you up will be ARGV.

In PHP ARGV works like this:

<?php
// test.php
echo $_SERVER['argv'][0] ?> 
$ php test.php 1234 
> test.php

C and BASH work the same way.

Ruby does things a different (PERL) way:

#!/usr/bin/env ruby 
# file test.rb 
puts ARGV[0] 
$ ruby test.rb helloworld 
> helloworld 

The difference is, in PHP, C, BASH the first element of ARGV is the program's name. In Ruby, and in PERL, it is the first argument passed into the program. 

I'm trying to think which makes more sense, probably the Ruby/PERL implementation. I'm used to starting at index 1 though. So just bear in mind that even though constructs may be similar between languages, there is some devil in the detail.

Actually, it is a little bit like clicking 'start' to shutdown oin windows. To uninstall a plugin / feature you need to go to the install new software screen. On a Mac it's found by highlighting the help menu and selecting "Install New Software".

In the resulting window, there's a link which lets you view already installed software.

Clicking that link brings you to the installation details page which (assuming there are no dependency issues) lets you uninstall unwanted features.

Uninstall the unwanted software and restart Eclipse and you're good to go.

This can manifest itself in a number of ways, but the most common, I think, is what happened with me

aaron ~/Development/ruby/testapp $ rake db:create (in /Users/aaron/Development/ruby/testapp) Couldn't create database for {"reconnect"=>false, "encoding"=>"utf8", "username"=>"dbuser", "adapter"=>"mysql", "database"=>"testapp_development", "host"=>"127.0.0.1", "pool"=>5, "password"=>"testpwd"}, charset: utf8, collation: utf8_unicode_ci (if you set the charset manually, make sure you have a matching collation)

Seems the mysql driver gets confused a little bit as when you install the MySQL Rubygem as directed by Rake, you link against the bundled OSX MySQL and not the Macports one.

The solution is to install the mysql gem as follows (uninstalling it first, if necessary)

sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/opt/local/bin/mysql_config5

Substitute the mysql_config5 path with your own macports mysql_config path. Afterwards, everything should work fine.

aaron ~/Development/ruby/testapp $ rake db:create (in /Users/aaron/Development/ruby/testapp)

// Retrieving documents with find() method using a query string
$query = $idFieldName . ':' . $docId;
$hits  = $index->find($query);
foreach ($hits as $hit) {
    $title = $hit->title;
    $contents = $hit->contents;
}

This proves _painfully_ slow, you're loading the full index in an attempt to find a unique document with an ID. Even worse is if your unique ID happens to be a string such as a url or path. Since ZF 1.5, the 'best practice' direction to perform this type of task is to use the Zend_Search_Lucene::termDocs() method:

$term = new Zend_Search_Lucene_Index_Term('/somepath/somewhere', 'path');
$docIds = $index->termDocs($term);
foreach ($docIds as $id) {
    $doc = $index->getDocument($id);
    $title = $doc->title;
    $contents = $doc->contents;
}

Performance wise this proves much more efficient. However, unless you're careful at the indexing stage you may run into trouble when running termDocs() on a string value such as a URL or path as opposed to an integer ID. This is down to the field being added tokenized. This is the most common way fields are added and corresponds to:

$doc = new Zend_Search_Lucene_Document();
$doc->addField(Zend_Search_Lucene_Field::Text('title', $title));

If you want to use termDocs on an identifying field you need to add the field as type Keyword:

$doc = new Zend_Search_Lucene_Document();
$doc->addField(Zend_Search_Lucene_Field::Keyword('http://a.com/uri', 'uri'));

Keyword fields are not tokenized, and a term vector (which termDocs() requires) is stored, the distinction between the two field types is documented in Zend_Search_Lucene_Field's phpdocs:

Zend_Search_Lucene_Field::Text() constructs a String-valued Field that is tokenized and indexed, and is stored in the index, for return with hits. Useful for short text fields, like "title" or "subject". Term vector will not be stored for this field.

In contrast see:

Zend_Seach_Lucene_Field::Keyword() constructs a String-valued Field that is not tokenized, but is indexed and stored. Useful for non-text fields, e.g. date or url.

This caught me out a little bit until I dug around the source a little bit looking to see where termDocs was going wrong. Hopefully this helps save someone else some time, and hopefully Zend can update their documentation to draw other developers' attention to this quirk.

If you are looking for the long $filters for ReflectionClass::getMethods(), here they are. They took me a  long time to find. Found nothing in the docs, nor google. But of course, Reflection itself was the final solution, in the form of ReflectionExtension::export("Reflection").

// The missing long $filter values!!!
ReflectionMethod::IS_STATIC;
ReflectionMethod::IS_PUBLIC;
ReflectionMethod::IS_PROTECTED;
ReflectionMethod::IS_PRIVATE;
ReflectionMethod::IS_ABSTRACT;
ReflectionMethod::IS_FINAL;

// Use them like this
$r = new ReflectionClass("MyClass");
// Print all public methods
foreach ($r->getMethods(ReflectionMethod::IS_PUBLIC) as $m) {
    echo $m->__toString();
}

Another example, this time one of my own, is one that I found myself writing while working with the Zend Framework's Zend_Controller implementation:

/**
 * @param   String $controller_class
 * @return   ArrayObject
 */
private function getActionList($controller_class)
{
    $reflection_class = new ReflectionClass($controller_class);
    $methods = $reflection_class->getMethods(ReflectionProperty::IS_PUBLIC);
    return new ArrayObject($methods);
}

Like many platforms, in PHP it seems documentation is no replacement for digging around the source itself.

Firstly you need to backup your forum database, the simplest way is to make use of the mysqldump utility.

mysqldump -uusername -ppassword forum_db_name > forum_db_backup.sql

Copy this output somewhere safe just incase of disaster. You can always revert to what you had by reimporting from this dumpfile. By default VBulletin applies the 'latin1' charset to its schema. We want to replace all instances of this in the dumpfile with 'utf8'.

sed -i 's/latin1/utf8/g' forum_db_backup.sql

Now it's time to take your forum offline. Navigate to: AdminCP -> VBulletin Options - > Turn your VBulletin on/off and change the forum active option to off. When this is done, just for good measure, 'chmod 000' the forum root directory.

Drop the old VBulletin forum database and create a new, utf8 friendly, one.

mysql --user=username --password=password --execute="DROP DATABASE forum_db_name; CREATE DATABASE forum_db_name CHARACTER SET utf8 COLLATE utf8_general_ci;"

It's time to convert the old database dump to use the UTF-8 character set. My favourite utility to do this is iconv. This util can easily be installed using yum, apt, emerge, synaptic or whatever package manager your distro relies on

iconv -f latin1 -t utf-8 forum_db_backup.sql > forum_db_backup-utf8.sql

Pump the fixed SQL back into your database.

mysql -uusername -ppassword forum_db_name < forum_db_backup-utf8.sql iconv -f iso8859-1 -t utf-8 vbulletin-language.xml > vbulletin-language-utf8.xml

Then import the language in the usual way AdminCP -> Languages & phrases -> Download/upload languages.

It's good to think of your CV as like a tool, no different from a programming language or an IDE. Like any tool it has to be rigorously tried, tested and possible refactored if it is to be successful and help you achieve your goals. In the case of a CV its purpose is two-fold, a) to get yourself past a recruiter and onto the actual client and b) to impress the client sufficiently that they want to interview you. So in the spirit of the new year and everyone having 'ten things' lists, here's mine for what to avoid in a modern CV for an IT role:

1. Don't send an old CV. If you haven't updated your CV in months, let alone years, it isn't going to sell you as effectively as it can. As a developer or even a recent graduate you're learning all the time. Detail and demonstrate as much of that learning (that is pertinent to the job at hand) as possible. As you get older, more experienced, your perception of what is and isn't important improves. While recently reviewing my own CV I found I had added a significant amount of detail, while actually reducing the overall bulk of the document.
2. Please, do not include an interests section. This point is somewhat subjective I grant. But of the CVs we've been forwarded all but one had an interests section. Not one of these contained anything relevant to the job being applied for, and in some cases actually contributed to us ruling the candidate out. When hiring, a client is seeking a professional. Telling them you spend a lot of time playing 'World of Warcraft' in your spare time isn't going to get them jumping out of their skin to interview you - particularly if you don't qualify how that interest is relevant to you performing the duties required.
3. Avoid sending a generic CV. Applying for jobs is tedious, time consuming and not immediately rewarding. But sending a generic CV detailing every subject you did at GCSE and A levels doesn't tell the client why they should hire you, and you specifically, for the job at hand. Oh and most clients seeking developers don't care if you worked at Subway or have a Tesco certification in fish mongering, it's simply not relevant.
4. Web 2.0 extravagance might be 'in' right now, but try to keep your CV conservative, well spaced and using a minimum of fonts. Giant mastheads, fancy bullets, a giant mess of fonts, it isn't impressing anyone, seriously, and most recruiters gut your CV into a standardised format anyway. Do be mindful of how it is to read. Clean and clear is the key, but save the extra special effort for the content.
5. Don't go on for longer than two pages. Trying to cheat this rule by making the font so small that a microfiche reader is required is only going to annoy the recruiter. That means your CV gets forwarded to the 'round file' not the client's inbox.
6. Don't list every course you did at University and expect the client to care. Of our graduate applicants not one has detailed what the courses actually contained. Database Systems 1. Cool - and you did what exactly in that course that is relevant to the job? It's often difficult for graduates to pad out their CV in the same way as someone with a few years' experience. Treat your degree like a job, work out what specific skills you got from your courses and detail the skill qualified with a practical example of how you applied it in your course work. Start doing that for all of your courses and very quickly you'll bang into the two page rule. Keep going, but when you're finished pare down the text into bullet points that are directly relevant to the job spec.
7. Spelling mistakes! Simply do not forget to proof read your CV. This really ought to go without saying. Get your sister, mother, girlfriend, house mate, anyone, to read your CV before sending it. Read it backwards, turn the zoom up to 400% and read it one word at a time. Anything to ensure that what you send out is as professional as you are. A sloppy CV leads to a sloppy impression. 'The Halo Effect', while not always fair, is something you have to deal within the real world. If a client's first impression of you is a bad one, all subsequent impressions will be shaded by it.
8. Don't include a picture of yourself. Like ever. It's simply not necessary.
9. Don't ramble, don't generalise, don't make statements without qualification. Good use of English is one of the best indicators of a good developer. Good code is succinct, clear and well qualified (unit tested). Like PHP, Java, indeed any programming language, English has a grammar, a vocabulary and patterns of use. Make best use of these attributes in your CV. Constantly refactor your CV to weed out redundancies, group like concepts, simplify and support the concepts. Like good code, a good CV goes through many iterations until it's 'good enough' for the real world. Mercilessly refactor your CV as you would your program code.
10. Don't accept everything you read on the net about CVs as gospel. This caveat applies to everything in life and not least this blog itself. If you listen to every top ten list of what not to include in your CV you'll quickly find out there's absolutely nothing you should put in your CV. Critically consider what you read suggested about what a good CV looks like and make your own mind up based upon the supporting arguments made and your own CV's feedback. For example if you disagree about point two and decide to include an 'interests section' ask recruiters when they call you what they think about it, did it provide value or was it noise? If you're getting interviews ask the recruiters what in your CV is standing out. If you're not, then ask what, if any, feedback there is from the client - and react to it.

I really do think of a CV like program code or a developer's utility. As a good developer you always need to evaluate your tools to ensure they are providing the maximum value. And like program code there is always an element of entropy to catch and counter over time. A CV, like program code, can sometimes be brittle and break with additions. Using and maintaining one is a matter of constantly reviewing, refactoring and responding to user feedback. For some good resources on CV writing I really recommend the following links: http://www.harveynash.com/uk/hnit/jobs/articles_cv_basics.asp http://www.lifeclever.com/give-your-resume-a-face-lift/ And for graduates especially: http://www.kent.ac.uk/careers/cv.htm

To guide me through the Ruby syntactic jungle is O'Reilly's 'Learning Ruby' book. I checked out the ubiquitous Pragmatic Programmer's 'Programming Ruby', but the third edition seems more suited to stopping doors than teaching Ruby.

Reading through the first few chapters of 'Learning Ruby' has left me a little nostalgic of the early 90s when I was going through my first programming book 'Learning Perl'. Maybe it's the title similarities but mostly it's that Ruby's syntax feels as slippery to me as Perl's did.

I don't mean that in an endearing way.

While I'm in love with blocks (or closures whatever you prefer) and Procs and Mixins (Traits), a simple thing scares me. Constants.

Constant, the word is pretty clear in its meaning. It is a value which remains stable over time. Wikipedia says: 'a constant is a special kind of variable whose value cannot typically be altered by the program during its execution'. Collins dictionary says: ‘something that does not or cannot change or vary’.

Ruby though, says no. Constants are just global variables declared capitalised. Oh sorry that should be capitalized. Ruby doesn’t speak English, only American. A constant in Ruby is about as constant as your discipline. If you REALLY want to change its value, Ruby wont stand in your way, or even make it that hard to do.

That really, really concerns me. Part of the appeal of Ruby, I guess, is that it lets you do things you want, your way. Maybe it’s ‘Fear’ but I really don’t want to have to worry about my constants changing. Ruby makes a big deal of its ‘duck typing’ You know, if it quacks like a duck, it is a duck. Unfortunately in Ruby a constant quacks like a duck but it bites like a hippo.

This is, of course, not absolute, there are camps of reformed Perl sys-admin monkeys that haven’t been seduced by Python. And of course a rare breed of ex systems programmers versed in C and C++.

It is actually this last group of coders that tend to get on with PHP the best.

This makes sense, because PHP was originally designed to be a template engine to produce HTML with a core C backend doing the grunt work. This seems a ridiculous idea now, as CPU cycles are much cheaper than man hours. But in the late 90s native code ran circles around dynamic, interpreted alternatives.

These days C is a little bit like ASM was in the 90s. Being proficient in it is handy, but not essential. I would argue though, for PHP coders it is an important environment in which to be familiar.

Many of PHP's eccentricities can be traced to it’s C roots. The strange and inconsistent function parameter orders, right down to the sometimes peculiar way it handles memory and references.

The file and string functions marry up almost 1:1 with standard C. Experience of and understanding of how these low level functions work will immeasurably improve your application of their PHP counterparts.

Networking is another area where PHP and C are very similar. Indeed you can write a simple tcpclient in PHP, get rid of all the $ variable identifiers and have a recognisable C fragment.

The biggest benefit though of learning a bit of C or C++, is to get an “appreciation” of memory management. C is unique among the programming environments of today in that it does not manage memory for you. Arguably this is what makes C so painful to develop in. But similarly, a sometimes useful characteristic that makes the environment still relevant today.

By way of example, strings in C are represented by an array of chars, terminated by a NUL ‘\0'.

In C, you would declare a string like  this: char str[20];

This declares a string of 20 characters, and str itself points to an address in memory where 20 bytes have been reserved for itself. Now if we try to write 21 characters to this string, C wont autoexpand the array to fit, or do anything so helpful as to warn you that you’ve gone out of the bounds of your allocated storage. No, C will do what you tell it, and in this case it means overwriting someone else's data.

This is quite a powerful characteristic and as such requires a programmer to be responsible.

Generally if you have a 20 char string defined but want to store a 25 char string, you need to reallocate memory. You could declare a new array or do a concatenation operation. Either way, this costs CPU time.

So when in PHP you are spamming '.' concat operators everywhere, you can appreciate that PHP is doing a lot of memory re-allocation under the hood to provide that syntactic sugar.

As mentioned above, PHP was originally intended to be a purely templating language for C web applications. Which is where PHP modules / extensions come in. These originally were where your business logic was to go. PHP extensions are compiled to native code, and as such run FAST. You can then access these functions in PHP just like the other core functions and classes. Yahoo, Facebook, use PHP in this way.

So can you. A great way to dabble in a bit of C is to write your own PHP extension. The benefit of getting your hands dirty in this way is the nice speed boost for the functions in your extension. But the more important benefit is how it can greatly improve your understanding of what PHP does for you under the hood. This lets you make informed design choices when writing your regular PHP code.

As a starting point, I can thoroughly recommend starting out with the Zend engine hacker's guide on the main PHP site. Becoming familar with the structure of the internals is the first step to appreciating all the of the magic PHP does for you under the hood.

But how does this affect you as a developer, and what is the reasoning behind each different approach. This paper goes a long way to explaining this, rather fundamental, aspect of computer science.

This can be inconvenient at times when you already have the data sitting in a variable. A way of getting around the need to worry about physically creating a file is to use the memory stream type.

PHP supports a number of input / output streams ranging from the usual stdin, stderr, stdout to memory, temp and filter.

See http://php.net/manual/en/wrappers.php.php for more information on these.

But looking at the memory type, it's very easy to use. Simply $fh = fopen('php://memory', 'wb+'); and you can use the usual file functions you would typically associate with an ondisk file.

You can fread, fwrite, file_get_contents on the memory stream or push it out over the network using the tcp streams PHP offers. PHP streams are a powerful and often underutilised aspect of the language.

SELECT DATE_FORMAT(yourdate, '%Y-%m') AS grouping_date COUNT(id) FROM yourtable GROUP BY grouping_date;

Change the date format to whatever date quantum you want to report on, i.e. just have '%Y' for a yearly grouping, or have '%y-%m-%d' for a daily one.

The quickest way, is actually very simple, and that is to simply use curl to resubmit the callback. This is assuming you have found, and if necessary corrected, the issue that stopped the callback succeeding in the first  place.

When a callback fails, either through a time out or some internal (500) error, you will receive an email from Worldpay notifying you. This mail includes two attachments one the request data Worldpay sent to your callback URL including the encoded post data as well as the response from your server.

Now assuming you work around your security arrangements (e.g your callback URL should listen only to certain addresses and expect auth credentials), you can use curl to resubmit your post data:

curl -d ‘testMode=0&authCost=716.86&currency=GBP&address=15+Somewhere+Street&countryString=United+Kingdom&callbackPW=xyz&installation=2555555&fax=&countryMatch=Y&transId=1000000000&AVS=2222&amountString=%26%23163%3B716.86&postcode=XX11+1XX&msgType=authResult&name=Mr+Test+Tester&tel=0208+111111&transStatus=Y&desc=Test+Transaction+1234&cardType=Visa+Delta&lang=en&transTime=1277741069650&authAmountString=%26%23163%3B716.86&authAmount=716.86&ipAddress=80.80.80.80&cost=716.86&charenc=UTF-8&instId=2555555&amount=716.86&compName=Testing Tester&_SP.charEnc=UTF-8&country=GB&rawAuthMessage=cardbe.msg.authorised&authCurrency=GBP&email=testy@tester.com&cartId=12345678&rawAuthCode=A&authMode=A’ \
'https://mysite.com/callback'

Now your program can handle the completion of the order as normal.

You can make this more sophisticated, by having a mail reader retrieve any missed payment notifications and parse them. With each failure notification you can add a retry to a job queue, or other batch processing stack. It would be worth adding some sort of retry threshold though, as well as sending notifications to developers to ensure there is no serious malfunction occurring.

For the impatient, the cli programme that weaves this magic is called 'textutil'.

But  anyway, using svn:externals to include external libraries is a convenient way of moving third party libraries out of your own repository.

For example, many PHP applications today, have the Zend Framework as a dependency. To include the library within your application without actually having to download the tarball and use something like phing to set it up in place, you can simply do a

svn propedit svn:externals

Then in your editor, add the following line (each externals definition goes on a new line)

Zend http://framework.zend.com/svn/framework/standard/tags/release-1.10.5/

Set this property on your standard application library directory, or somewhere in your include path.

When you run svn update, svn will pull down the Zend library into your defined location.

The Zend Framework, out of the box, comes with some dojo dependencies that are optionally bundled with the framework. You can use the same approach to pull in the dojo deps by setting svn:externals definitions for your javascript include dir and pointing them at http://svn.dojotoolkit.org/src/tags/release-1.4.3/ (or trunk if you're brave).

Again, there are some good arguments for NOT doing this, but for me, the convenience of this approach outweighs those (which I'll cover in more detail later).

The first one is the one you're most likely to encounter after upgrading

 Fatal error: Call to a member function toHtml() on a non-object in <somewhere>/app/code/core/Mage/Core/Model/Layout.php on line 52

The solution is explained here:

http://rackspeed.de/forum/magento-faq-installation-and-updates/fatal-error-call-to-a-member-function-tohtml-on-a-non-object-layout-php-on-line-529-a-547

Basically edit your theme's layout/page.xml and change <block type="core/profiler" output="toHtml"/> to <block type="core/profiler" output="toHtml" name="core_profiler"/>

The next two are frankly amazing in that they ever made it into the release.

See here:

http://www.magentocommerce.com/boards/viewthread/195761/

To quote the finder of these issues 'Yes, preparing a shipment prepares an invoice instead..'

In short, when overriding a helper, the context element IS case sensitive. With blocks, it is NOT.

<![CDATA[
{{depend prefix}}{{var prefix}} {{/depend}}{{var firstname}} {{depend middlename}}{{var middlename}} {{/depend}}{{var lastname}}{{depend suffix}} {{var suffix}}{{/depend}}|
{{depend company}}{{var company}}|{{/depend}}
{{var street1}}|
{{depend street2}}{{var street2}}|{{/depend}}
{{depend street3}}{{var street3}}|{{/depend}}
{{depend street4}}{{var street4}}|{{/depend}}
{{depend city}}{{var city}},  {{/depend}} {{var region}} {{depend postcode}}{{var postcode}} {{/depend}}|
{{var country}}|
{{depend telephone}}T: {{var telephone}}{{/depend}}|
{{depend fax}}<br />F: {{var fax}}{{/depend}}|
]]>

The format may be slightly terse but hopefully you can see how it can be applied to create your own desired format.

With magento models, you may be tempted to use camelcasing for long class names, for example MyPackage_MyModule_Model_ALongNameForAModel. This will not work as you might expect on different (namely case sensitive) environments.

Your Mage::getModel(mymodule/a_long_name_for_a_model) call will translate to the class loader trying to find A/Long/Name/For/A/Model.php.

Conversely trying to address your model Mage::getModel(mymodule/alongnameforamodel) will see the classloader trying to load Alongnameforamodel.php, and not ALongNameForAModel.php.

On Windows this is fine, on case-sensitive e.g., HFS (Mac) or Unix file-system, this will not work.

UPDATE cms_page SET custom_theme = 'custom/default' where custom_theme='default/custom';

As one of my clients' sites has grown, some batch updates were taking up to 30 minutes to run. This is too long.

If the changes you are making are just to update simple attributes, for example we have a sales ranking attribute, you can use the following code to update the value without incurring the massive overhead of a full product save.

$product->setNumSales(1234); $product->getResource()->saveAttribute($product, 'num_sales');

The saveAttribute method takes two parameters, the first is the model containing the attribute value, the second the attribute code. To find out the attribute code, look it up in either the db (eav_attribute) or in the admin backend under catalog->attribute.

Using the getResource()-->saveAttribute() call, takes 1/5s, doing a full save(), takes 2-3 seconds. When iterating over a large product base, that is HUGE.

Update 4 Mar 2014 - Please take a look at DannyD's comment below for a more robust approach to mass attribute updates.

For example, say you want to select all records from a table that has a date set more than 1 month old (compared to the current time).

SELECT * FROM table WHERE DATE_SUB(CURDATE(),INTERVAL 30 DAY) > date_column

This assumes your date column is of type date, or datetime, although using FROM_UNIXTIME will allow you to work with timestamps as well.

The date_sub call, if ran on 2010-05-20, will return 2010-04-20. So if your date_column field has a date greater than this value, it is therefore more than 30 days old.

This is particularly handy when generating reports.

I just saw an insane suggestion on a Magento forum, advising users to change their admin url by hacking app/code/core/Mage/Adminhtml/etc/config.xml.

This is dangerous and pointless. Dangerous, because when you attempt to upgrade magento, you will at best lose your custom configuration, and at worst, cause an upgrade to fail. Pointless, because you can achieve the same effect by making changes to app/etc/local.xml.

This file overrides any configuration set elsewhere and by default, actually already has a definition for a custom admin url. When you install Magento, the installer sets up this file up by subbing in the details you provide the installer into local.xml.template.

So to change your admin url, after running the installer, simply fire up an editor and open app/etc/local.xml and look for this:

<admin>
    <routers>
        <adminhtml>
            <args>
                <frontName><![CDATA[admin]]></frontName>
            </args>
        </adminhtml>
    </routers>
</admin>

Change the admin (inside the CDATA declaration) to foobar, or gobbles, or topsecret, whatever. Much easier, much safer than hacking away at core magento configuration files.

To add a text template to a view in magento, you need to consider three aspects

1. The template block itself
2. The block, or layout, the template will sit in
3. The layout .xml configuration

The first element is simplest, create some html, stuff it into a .phtml file and copy it to a directory within your theme, which resides (relative to the store root dir) in 'app/design/frontend/<package>/<theme>/templates'. I'm going to assume the package/theme is default/default from here forward.

You will now need to configure one or more layout xml templates (app/design/frontend/default/default/layout') to refer to your new template block. For most general purpose, globally available blocks, this will be 'page.xml'.

For example, to add a productfinder template to the three column layout, you need to edit page.xml and within the

<block type="page/html" name="root" output="toHtml" template="page/3columns.phtml">
...
</block>

block add the following code near the closing </block>:

<block type="core/template" name="my_product_finder" as="my_product_finder" template="templatedir/productfinder.phtml"/>

Type="core/template' refers to a Magento class Mage_Core_Block_Template. This type can be different if you want a specific type of block. But core/template is the simplest way to include some text on a page. The 'name' and 'as' attributes allow you to reference the block in your enclosing templates, e.g. 3column.phtml, or other layout .xml files. The template attribute is the relative path to the text template you want to include and is relative to the theme template root i.e. app/design/frontend/<package>/<theme>/templates.

Once you have created a layout definition for your block, including the template on a page is easy. In 3column.phtml, for example, simply put <?php $this->getChildHtml('my_product_finder') ?> and the text will be included. Remember to refresh, or clear Magento's cache first though.

One more element to consider is exclusions. If you put the above code in 3column.phtml, that text will appear on every page that uses that layout template. You may not, for example, want to include it on customer dashboard pages. In this case  you will need to define a remove statement within the layout.xml for the group of pages you want to change. In this example, we need to edit customer.xml

In customer.xml (and indeed page.xml) you will see a number of elements similar to <default></default> (which applies to ALL pages), <customer_account></customer_account> (which refers to all pages with the url customer/account/), and <customer_account_edit></customer_account_edit> (which refers to all pages with the url customer/account/edit/). Including your remove declaration in the right element here allows fine grained control over which pages your blocks appear.

In the example here, we want to remove the product finder from ALL customer account pages. Therefore within the <customer_account> element we add the code

<customer_account>
...

<reference name="root">
<remove name="my_product_finder"></remove>
</reference>

</customer_account>

The reference name=root element ensures we are altering the root block, which is the one in which we defined our template block in page.xml. The enclosed remove call, uses the name of the block we created "mns_product_finder" to identify the block we want to remove for these pages.

That's it! All pages that exist under <customer_account> (so all of them) have this block removed.

To see if you are affected run memcached in -vvv mode and look out for this:

CLIENT_ERROR bad command line format.  Usage: delete <key> [noreply]

This causes magento to throw an exception with the message ""can't get stat from localhost:11211".

The problem is down to a change in behaviour of memcache in version 1.4 where they removed the second parameter passed to the delete function. See here for more info http://code.google.com/p/memcached/wiki/ReleaseNotes144. 1.4.4 thankfully added some backwards compatibility.

Magento 1.4 uses Zend Framework 1.9.6 and this, sadly, does not support these changes. Version 1.10.3+ does.

http://framework.zend.com/issues/browse/ZF-9376

I use iTerm as my terminal of choice, and it natively supports utf-8. However by default, your terminal environment does not. I have my environment configured to use the de_DE (German) locale. However some characters, most typically ü and ä do not display correctly.

In order to get correct text in your utf-8 terminal you need to configure your environment to use a utf-8 enabled locale. If you're using en_GB, or de_DE (and you can get a list of available locales by calling 'locale -a') you just need to edit /etc/profile or ~/.profile or ~/.bash_profile, and put the line export LC_ALL='de_DE.UTF-8' or LC_ALL='en_GB.UTF-8'

This will ensure your terminal AND environment are speaking the same language.

This happens when you try to render non utf-8 text as utf-8. Utf-8, shares the same 7-bit ASCII range as common windows encodings cp-1252 and iso-8859-1. Lower ASCII is the basic alphabet e.g. a-z, A-Z, hyphens, commas, etc. However extended characters, like accents, symbols and umlauts change. So a cp-1252 trademark symbol has a different code in utf-8. Try to render a cp-1252 copyright on utf-8 you will just see a question mark in the browser, as that code does not have a corresponding entity in utf-8.

To fix this you need to normalise your content. If it's going into mysql, ensure your tables are set to your normalised format. I recommend using utf-8 as this normal form.

To normalise your text, use iconv, which is available as a php extension.

$utf8Text = iconv('iso8859-1', 'utf-8', $isoText);

When outputing to the browser run this through htmlentities or htmlspecislchars. Both these functions expect iso-8859-1 input. To correctly prepare your utf-8 text for output you need to supply a third parameter to these functions supplying your text's encoding, in this case utf-8.

htmlentities($text, null, 'utf-8');

Failing to supply this parameter means your text will be broken and your extended characters displayed as questiomarks.

PHP provides all the tools you need to properly handle many different text encodings. As a developer you just need to normalise your input, and properly encode your output.

From the man and help pages it would seem doing sed -i 's/hello/goodbye/g' helloworld.txt would be the way to achieve it.

It doesn't, and you'll either get a script processing error or something like: sed: -i: No such file or directory.

The trick, is to use it like this: sed -i '' 's/hello/goodbye/g' helloworld.txt 

Still, it's very easy to get the baseurl directly from magento, below I've summarised the most common ones you'll want.

Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_WEB);
http://myshop.mydomain.com/

Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_LINK);
http://myshop.mydomain.com/index.php/

Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_JS); http://myshop.mydomain.com/js/

Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_MEDIA);
http://myshop.mydomain.com/media/

Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_SKIN);
http://myshop.mydomain.com/skin/

So, say you want to create a link to your store's checkout page you would use the following code:

<a href="<?php Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_WEB);?>checkout/onepage/">Checkout</a>

It's well worth taking a look at the getBaseUrl function in the Mage class(app/Mage.php) and Mage_Core_Model_Store (app/code/core/Mage/Core/Model/Store.php)

I had an array of files that I wanted to do some operations on and the following construct allowed me to easily iterate through that list.

FILES=( a/path/to/a/file1 a/path/to/another/file2 and/so/on/and/so/on )

for ELEMENT in $(seq 0 $((${#FILES[@]} - 1))); do
  echo ${FILES[$ELEMENT]}
done 

The $(seq 0 $(($#FILES[@]} - 1))); returns the number of elements in FILE, the seq command produces a sequence of numbers from x to y. If you call seq 0 4, you will get a line with 0, 1, 2, 3, 4 on it.

So while the syntax is a little smelly, the terse power of it, is quite handy.

Often, you will want to change some details, baseurls, use test payment or shipping account details.

Generally this is pretty easy to script up in bash, python or even plain php. However magento encrypts some of the data it stores in core_config_data.

The following is the approach I use to update data in core_config_data  in an encrypted format magento will accept:

<?php
require_once 'app/Mage.php';
umask(0);
$app = Mage::app('default');
$data = $_SERVER['argv'][1];
$obj = Mage::getModel('core/encryption');
$helper = Mage::helper('core');
$obj->setHelper($helper);
echo $obj->encrypt($data);

I use it as part of my git workflow (see my magento githooks repo) and I find it really does help keep code consistent.

When you work on projects with more than a couple of developers, niggling differences in style can lead to unintentional errors creeping into your codebase (think omitting braces in if statements, for example).

Anyway, one of the neat features in the master branch is the ability to specify a config file to control how php-cs-fixer behaves rather than having to define everything on the commandline (or depend on .php_cs being in the path). Unfortunately this feature is not yet in a pre-baked release of php-cs-fixer. So to use it you have to use the source version. I quite like the convenience of the phar version but it's unclear how to build one directly from the sources.

After a bit of digging around in the code and issues list, I found they are using the php-box project to build release phars. It's actually very simple, but to save others having to figure it all out, just follow these steps.

$ git clone https://github.com/fabpot/PHP-CS-Fixer
$ cd PHP-CS-Fixer
$ composer.phar require --dev 'kherge/box=~2.4'
$ vendor/kherge/box/bin/box build
> Building...

Do a quick ls and you'll notice you have a minty fresh php-cs-fixer.phar file.

Done.

$ usermod -a -G thegroup theuser

And job done. But OSX uses Open Directory rather than traditional flatfiles like /etc/passwd and /etc/group to store information about users and domains. So the typical unix commands we are used to, don't work.

The dscl (directory service command line) utility lets you manipulate Open Directory values and in our case, add a user to an additional group, handy if for example, you want to add your user to the wheel group to make use of password free sudo.

$ dscl localhost --append /Local/Default/Groups/<groupnamehere> <usernamehere>

So, later on, you'll probably want to pull changes down from the remote and you'll end up seeing something similar to this

➜  store git:(zendesk) git pull --rebase                                                          [48/53]There is no tracking information for the current branch.
Please specify which branch you want to rebase against.
See git-pull(1) for details

git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

git branch --set-upstream-to=origin/<branch> zendesk

Now it's not that hard to type out the suggested command above to set the upstream branch, but I got sick of having to do it so often, and I have given up trying to remember -u, so I created a git alias to automate things and save some keystrokes.

In your ~/.gitconfig under the alias section, add this

    sup = !git branch --set-upstream-to=origin/`git symbolic-ref --short HEAD`

You can use the alias by issuing the following command in your terminal

$ git sup

This will look at the current branch and set its upstream to origin/branchname

If you tend to use another remote name other than origin, change the alias accordingly.

I have a few other useful aliases which you can checkout (hah, sorry :)) in my full gitconfig.

For some workarounds checkout http://mywiki.wooledge.org/BashFAQ/024

Usually these types of commits can be avoided by keeping feature branches up to date with git --rebase. When two branches have a direct common history, merges can be applied using the fast-forward strategy avoiding the need for a stitch-things-together merge commit.

Because the commit pointed to by the branch you merged in was directly upstream of the commit you’re on, Git moves the pointer forward. To phrase that another way, when you try to merge one commit with a commit that can be reached by following the first commit’s history.

http://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging

To ensure you've kept your branches synced up with rebase and to avoid accidentally creating a merge commit, you can set git merge to only perform fast forward merges.

$ git config --global merge.ff only

This way, you'll get a gentle reminder to rebase. If that's not feasible then you can force through the merge with

$ git merge --no-ff

• grumpy-phpunit.com
• Price: $29.00

The Grumpy Programmer (actual name Chris Hartjes) amusingly blogs and tweets all things PHP and particularly PHPUnit. When I saw he was publishing this book, I was curious to see how his strident style would stand up to the longer form. Pleasantly, it turns out.

Chris maintains his gruff voice while whirling through the ins and outs of using PHPUnit. I've been using PHPUnit for a long time now and I find that when I am really familiar with a tool, I tend to re/(over)use patterns that have served me well in the past. So while I found I was familiar with much of the material in the book, there are more than a few tidbits here that I picked up. I feel even the most grizzled PHPUnit veteran's testing regime will benefit from a read through.

The book seems aimed at the less experienced which I did find a little surprising given the title. When I think cookbook I tend to think of the weighty O'Reilly tomes. This book though is more like a lengthy tutorial than a cookbook in the O'Reilly style. As a tutorial goes though it excels. It is detailed without being turgid and covers all the major aspects of using PHPUnit that I would expect it to and then some. I found the chapter on Test Doubles (that is mocks, stubs and fakes) to be particularly excellent. The vocabulary surrounding these terms tends to get mixed up and consequently programmers often treat them as the same thing. That leads in my experience, at best, to confusion. And at worst, to poor tests that are difficult to maintain.

As a quick aside, the book is published by LeanPub, who ensure authors receive 90% of the proceeds from their work. I think this is a wonderful initiative. Writing, especially for a programmer is tremendously hard and I like the idea of those that attempt it, and do a good job, get appropriately rewarded for doing so.

So, back to the book. You find peppered throughout the introduction to PHPUnit subtle wisdoms that are hard to argue with. A simple and you would think obvious example, is that of always providing the final argument to assert statements with a description message. This message is displayed when the test fails, helping you quickly identify where the problem lies. Another: writing strictly encapsulated code that eschews static methods and class variables, is (well unsurprisingly) easier to test that code that is constantly mutating global state.

The book is quite short, coming in (at least in my pdf version) at 85 pages. I feel like there is sufficient scope for more content here. Especially for a 'cookbook'. I would have loved to have seen more on using Data Builders for example. The chapter on data providers is great, but I find you often need more fine grained control over your fixtures. Factories and data builders are a couple concepts that once learned, significantly reduce the friction of TDD.

I perhaps would also liked to have seen more in introduction to TDD itself, motivations for it, and perhaps a brief comparison between the two principle TDD xUnit styles. Specifically the Statist TDD and Mockist/London School TDD styles. The former being a test style mainly interested in setting up some state, running a behaviour and checking the end state matches what you expected. The Mockist approach is less interested in observing State and instead is more interested in the messages passed between objects (method calls between collaborators).

Overall I enjoyed the book, and it fills a much needed role in guiding budding PHP TDD practitioners in the use of the most mature tool we have available in PHP. I picked up a few neat new tricks and I suspect many PHP programmers will do the same.

You can buy it now at grumpy-phpunit.com

The problem: you want to select the smallest value from a set of values.

Let's say you have a table of products that are in a logical group and you want to select the lowest priced product from that group, however some products actually have a 0.00 price (for whatever reason). You don't want to actually show 0.00 as the lowest price for this group of products, you want to show the lowest price that happens to be greater than zero.

MySQL has a neat way to do this. Simply go:

SELECT tableref.group_id, MIN(NULLIF(tableref.column, 0)) as min_price FROM tableref GROUP BY tableref.group_id;

The magic is in the NULLIF function, which will return null if tableref.column is equal to 0. Returning null removes that value from inclusion by MIN, having the effect of forcing the column value to be greater than zero.

So I'm happy with a less flexible, but attractive, functional and predictable desktop and I think many others feel the same way. It's no real surprise to me then, that OSX has mostly killed off the idea of Linux on the Desktop.

But somewhere that OSX falls severely behind, is its use of a BSD inspired Unix implementation. If you're born and raised on a diet of GNU (file|core)utils, of apt, yum, and portage, heck even sysvinit, OSX's realisation of Unix leaves a lot to be desired.

With considerable effort and some patience though, OSX can be brought to heel. With Iterm2 and Macports you can have a functional GNUlike Unix experience.

I'll go over the minutiae of my Macports setup another time, but generally speaking I replace all the default OSX tools with GNU equivalents and favour /opt/local/bin over everything else. It means I can have one set of configs which work mostly unchanged across Linux and OSX instances.

Macports is pretty good and the folks that contribute to it do a great job. But it does lack the polish that you take for granted with the Linux package managers. Another point to keep in mind is Macports, like Portage and BSD Ports, is a source-code based 'package' manager. When you install something, it is compiled right there and then on your system. When things go wrong, unless you're a competent C programmer (and even then) you're going to have a bad time.

One last thing to remember too, is OSX defaults to a case insensitive (but thankfully case-preserving) HFS filesystem. By default, PHP and php appear as the same thing to HFS.

So the point of this blog is to go over getting PHP running natively with Macports and how we can run an instance of Magento and the Magento Test Automation Framework (TAF).

MySQL

MySQL is probably the easiet part of the whole thing to setup. So let's start there. For reference, the database files are stored under /opt/local/var/db/mysql55.

In Macports MySQL carrys a namespace of sorts by the way of a version suffix (as does PHP). This lets multiple versions of a package be installed side-by-side. The drawback is rather than having a mysql command, you have a mysql55 command. That's annoying. So we will install mysql_select which lets us select a version to activate and give us proper file names.

$ sudo port install mysql55-server mysql55 mysql_select
$ sudo port select mysql mysql55
$ sudo port load mysql55-server

We will want a database for our magento application.

$ mysqladmin -uroot -p create magento 

PHP / PHP-FPM

Now we want to install PHP, PHP-FPM and the extensions Magento and TAF require.

$ sudo port install php54 php54-fpm php54-curl php54-APC php54-gd php54-pcntl php54-gd php54-mcrypt php54-iconv php54-soap php54-yaml php54-xdebug php54-openssl php54-mysql php54-pear php_select pear-PEAR

$ cd /opt/local/etc/php54
$ cp php-fpm.conf.default php-fpm.conf
$ cp php.ini-development php.ini

$ sudo vim php.ini
# set date.timezone and cgi.fix_pathinfo = 0

$ sudo vim php-fpm.conf
# make any changes for min / max num servers, error logging etc

The MySQL extension needs a little bit of prodding to look in the correct location for mysql.sock

echo 'pdo_mysql.default_socket=/opt/local/var/run/mysql55/mysqld.sock' | sudo tee --append /opt/local/var/db/mysql.ini

Once PHP-FPM is installed and configured you can use Macports to tell launchd to start it automatically.

$ sudo port load php54-fpm

PHP-Select

As with MySQL, Macports lets you install multiple versions of PHP side by side. This can be handy if you want to run PHP 5.3 and PHP 5.4 at the same time. I just install a single version, but Macports effectively namespaces everything. So rather than '/opt/local/bin/php' you have '/opt/local/bin/php54'. PHP Select, which we installed earlier fixes this by effectively 'activating' one version and creating the usual executable names we're accustomed to.

$ sudo port select php php54 

PEAR

PEAR is the single biggest pain in the whole process. And with some research it turns out its because Macports PEAR isn't even meant be used by end users (WAT?!).

There is no MacPorts port that installs the pear package manager application with the intent that it be used by the end user outside a MacPorts port install. If you want to use pear manually on your own then you should install it using gopear, composer or some other method. http://trac.macports.org/ticket/37683

So this goes a long way to explaining why Macports doesn't set PEAR up with sane defaults, or even put the pear command in the default path. But we can sort this all out easily enough ourselves.

$ sudo pear config-set php_bin /opt/local/bin/php
$ sudo pear config-set php_dir /opt/local/lib/php/pear
$ sudo pear config-set ext_dir /opt/local/lib/php54/extensions/no-debug-non-zts-20100525
$ sudo pear config-set bin_dir /opt/local/bin
$ sudo pear config-set cfg_dir /opt/local/lib/php/pear/cfg
$ sudo pear config-set doc_dir /opt/local/lib/php/pear/docs
$ sudo pear config-set www_dir /opt/local/lib/php/pear/www
$ sudo pear config-set test_dir /opt/local/lib/php/pear/tests
$ sudo pear config-set data_dir /opt/local/lib/php/pear/data
$ echo 'PATH=$PATH:/opt/local/lib/php/pear/bin' >> ~/.bashrc # or zshrc if you use zsh

Another issue you'll possibly have with PEAR, is it will default to the system PHP executable (/usr/bin/php) rather than your active Macports one. The pear command does test for an environment variable so we can set up an alias to pass this variable to pear on invocation.

Add an alias to your bashrc/zshrc in the form:

alias pear='PHP_PEAR_PHP_BIN=php pear'

Reload your bashrc/zshrc.

$ source .bashrc (or source .zshrc)

Now the alias is active we can check that it's working

$ /opt/local/lib/php/pear/bin/pear version
PEAR Version: 1.9.4
PHP Version: 5.3.15
Zend Engine Version: 2.3.0
Running on: Darwin avalanche 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

$ pear version
PEAR Version: 1.9.4
PHP Version: 5.4.12
Zend Engine Version: 2.4.0
Running on: Darwin avalanche 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

Now to make installing PEAR packages easier I turn the channel autodiscovery option on, which means you don't have to manually add channels for package dependencies (which there are a lot when installing phing or phpunit…)

$ sudo pear config-set auto_discover 1

Now add phing and phpunit and install them with all their optional dependencies and some extra packages for the Magento TAF.

$ sudo pear channel-discover pear.phing.info
$ sudo pear channel-discover pear.phpunit.de
$ sudo pear channel-discover pear.symfony-project.com
$ sudo pear install --alldeps phing/phing 
$ sudo pear install --alldeps phpunit/phpunit
$ sudo pear install phpunit/PHP_Invoker
$ sudo pear install phpunit/PHPUnit_Selenium
$ sudo pear install -f symfony/YAML

PECL/Extensions

Macports by default creates .ini files to load extensions in /opt/local/var/db/php54. If you manually build any extensions, add the appropriate ini file here, for example:

$ echo 'extension=yaml.so' | sudo tee /opt/local/var/db/php54/yaml.ini

Nginx

Apache/Nginx. It doesn't really matter. Both are great, but in production I use Nginx so I use it in development too. I install it with just the ssl extension enabled, to see the full range of available options, use:

$ sudo port variants nginx 

To install:

$ sudo port install nginx +ssl
$ cd /opt/local/etc/nginx
$ sudo cp fastcgi.conf.default fastcgi.conf
$ sudo cp fastcgi_params.default fastcgi_params
$ sudo cp mime.types.default mime.types
$ sudo cp nginx.conf.default nginx.conf
$ sudo mkdir conf.d sites-available sites-enabled ssl

Once installed, Nginx requires a little bit of work to hook up to PHP and particularly to work well with Magento.

$ sudo vim nginx.conf  
# Insert the following towards the bottom of the file (but inside the http block) 
map $scheme $fastcgi_https {
   default off;
   https on;
}

##
# Virtual Host Configs
##
include conf.d/*.conf;
include sites-enabled/*;

For each app just add a server block to sites-available, then symlink it to sites-enabled.

$ sudo vim sites-available/magento.dev.conf
# ...     
$ cd sites-enabled
$ sudo ln -s ../sites-available/magento.dev.conf 001-magento.dev.conf

This is the server block definition I use for magento development, feel free to modify it for your needs.

server {
    listen 80;
    listen 443 ssl;
    
    ssl_certificate     ssl/magento.dev.crt;
    ssl_certificate_key ssl/magento.dev.key;

    server_name magento.dev;
    root /Users/aaron/Sites/magento;

    location / {
        index index.html index.php; ## Allow a static html file to be shown first
        try_files $uri $uri/ @handler; ## If missing pass the URI to Magento's front handler
        expires 30d; ## Assume all files are cachable
    }

    ## These locations would be hidden by .htaccess normally
    location /app/                { deny all; }
    location /includes/           { deny all; }
    location /lib/                { deny all; }
    location /media/downloadable/ { deny all; }
    location /pkginfo/            { deny all; }
    location /report/config.xml   { deny all; }
    location /var/                { deny all; }
    location /shell/              { deny all; }

    ## Disable .htaccess and other hidden files
    location ~ /\. {
        deny all;
        access_log off;
        log_not_found off;
    }

    location ~ \.php$ { ## Execute PHP scripts
        if (!-e $request_filename) { rewrite / /index.php last; } ## Catch 404s that try_files miss

        expires        off; ## Do not cache dynamic content
        fastcgi_intercept_errors on;
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_param  HTTPS $fastcgi_https;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        fastcgi_param  MAGE_RUN_CODE default; ## Store code is defined in administration > Configuration > Manage Stores
        fastcgi_param  MAGE_RUN_TYPE store;
        proxy_read_timeout 120;
        proxy_connect_timeout 120;
        include        fastcgi_params; ## See /etc/nginx/fastcgi_params
    }
    
    location @handler { ## Magento uses a common front handler
        rewrite / /index.php;
    }
}

We've said our application lives on a server called 'magento.dev'. So let's tell our hosts file about that.

$ vim /etc/hosts
# Insert or append to an existing line
# 127.0.0.1 localhost magento.dev

Last thing that needs to be done is setting up a selfsigned ssl certificate / key pair and storing them under /opt/local/etc/nginx/ssl

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myserver.key -out myserver.crt
$ sudo mv myserver.key /opt/local/etc/nginx/etc/ssl/magento.dev.key
$ sudo mv myserver.crt /opt/local/etc/nginx/etc/ssl/magento.dev.crt

Once that's done, we can start nginx.

$ sudo port load nginx

Web App Directory Config

I keep my web apps living under /Users/aaron/Sites, but remember that every directory element in the path needs to have the executable bit set for all users (so the web server can traverse the directory tree). Literally this is a case of:

$ chmod a+x /Users/aaron && chmod a+x /Users/aaron/Sites

Install Magento and TAF

N98 Magerun is the coolest thing to happen to Magento development since well, I can't remember. It singlehandedly relegated a few thousand lines of cobbled together bash script to the bin.

$ cd /Users/aaron/Sites
$ curl -O magerun.phar https://github.com/netz98/n98-magerun/raw/master/n98-magerun.phar
$ chmod a+x magerun.phar
$ ./magerun.phar install
# Follow the directions and install to /Users/aaron/Sites/magento with base url http://magento.dev and database name 'magento'.

After all that work, hitting http://magento.dev should now bring up the magento demo store!

I've been playing with Magento's Test Automation Framework and it was the motivation for finally getting everything working properly natively.

TAF runs at a glacial pace and in my normal development environment (VirtualBox over NFS), the universe would have undergone heat death long before the TAF suite completed its running.

Unfortunately the documentation for TAF is a bit of a mess (I'll write about my experience with it soon), but what it offers - 1500 automated tests - is a pretty big attraction.

Installation is actually pretty easy. I am assuming you don't have git already installed (remember you can use port variants to see what extension options are available):

$ sudo port install git-core +bash_completion +credential_osxkeychain +doc +pcre +python27
$ sudo port install git-extras
$ cd /Users/aaron/Sites
$ git clone https://github.com/magento/taf taf
$ cd taf # /Users/aaron/Sites/taf
$ cp phpunit.xml.dist phpunit.xml
$ cp config/config.yml.dist config/config.yml
$ cd .. # /Users/aaron/Sites
$ curl -O selenium-server.jar http://selenium.googlecode.com/files/selenium-server-standalone-2.31.0.jar

To run the test suite open up a new terminal

$ cd /Users/aaron/Sites
$ java -jar selenium-server.jar

Now the test suite is good to go

$ cd /Users/aaron/Sites/taf
$ ./runtests.sh

The test suite takes a loooooong time, so go for a run or something.

Hopefully these steps help out other PHP developers suffering from OSX.

SQLSTATE[HY000] [2002] No such file or directory.

You fix it by just appending the sock file for the mysql version you're using into the PHP mysql.ini file. I use mysql55 so to fix PHP I do this

echo 'pdo_mysql.default_socket=/opt/local/var/run/mysql55/mysqld.sock' | sudo tee /opt/local/var/db/mysql.ini

$ sshfs -o ssh_command="ssh -i ~/ssh_keys/aaron@awshost.pem" aaron@aws.instance.com:/var/www/ ~/Sites/awshost

Actually, right after I posted this, I realised there's a better way to do it. That is, use the 'IdentityFile' option instead (as per the format of .ssh/config).

$ sshfs -o "IdentityFile=~/ssh_keys/aaron@awshost.pem" aaron@aws.instance.com:/var/www/ ~/Sites/awshost

If you have any problems then add '-o debug' to the above command to help track it down.

There are only two hard things in Computer Science: cache invalidation and naming things.

-- Phil Karlton

I wanted to grab the last bit of a url that I knew would be the name of an image. I knew strstr well but that operates by giving you the remainder of a string that occurs after some needle in a string haystack. I wanted this behaviour, but only from the last instance of the needle.

strstr — Returns part of haystack string starting from and including the first occurrence of needle to the end of haystack.

strrchr -This function returns the portion of haystack which starts at the last occurrence of needle and goes until the end of haystack.

Let's look at how these work with an example

$url = 'http://www.google.com/a/b/c/d,img';
echo strrchr($url, '/'); // prints /d.img  
echo strstr($url, '/');   // prints //www.google.com/a/b/c/d.img

Now I've been programming in PHP for pushing on 12 years and this one still did my head in. The names of two very similar behaving functions bare little resemblance to each other.

At this point the arguments and criticisms over the core API have been exhausted and there's little that can/will be done. But I do wonder if it would be worth creating an object library to encapsulate primitive functions such as String, Integer, Array, Float etc., I'm not sure how possible auto-boxing is with PHP, and indeed if it's even a good idea. But definitely some object wrappers would help ease this API pain.

The best way to predict the future is to invent it. -- Alan Kay

Recently I have written a little bit about Smalltalk, and in my enthusiasm I got hold of a book called Dealers of Lightning by Michael Hiltzik. It covers the rise and fall of Xerox's Palo Alto Research Center (PARC), the research center from which Smalltalk emerged.

I initially read it to learn more about the context in which Alan Kay imagined Smalltalk and to find out who was the Executive-X he mentioned in The Early History of Smalltalk (it was Jerry Elkind). However I ended up coming away with a lot more. In particular, a new appreciation for a number of scientists I previously knew very little about. Scientists that are almost single handedly responsible for the shape of modern computing.

I grew up in the 80s so I have no real personal appreciation for computing as it was before say, 1984. I had an IBM clone (an Amstrad) and a VIC20. At school we have Commodore 64s/128s to play with. So for me, a computer has always been something that sits on your desk, you turn it on and you type away and stuff pops up on the screen. But right up until the late 70s this paradigm was considered absurd. Wasteful even. It took the vision of Alan Kay and the technical genius of Chuck Thacker and Butler Lampson along with the almost unlimited cash of Xerox to realise.

The book opens up by transporting the reader back in time to the late 60s and lays out the genesis of PARC. It then proceeds in roughly chronological order with each chapter focusing on one of the scientists and/or their inventions. The book closes by looking at some of the reasons why Xerox couldn't transform its research into viable products. Nominally the story is about what Xerox PARC did, however Hiltzik couches everything in terms of the scientists and it is his ability to bring these characters to life that makes the book so riveting to read.

One of the most striking individuals of the story is the Impressario Robert Taylor, a man who as much as anything can be considered the grandfather of the Internet (nee ARPANET). The Kays, Thackers and Lampsons of the story are the geniuses but genius needs direction and at times support. This is the role Bob Taylor played. The story of PARC for better and worse revolves around him and his relationship with the researchers that shared his vision of interactive computing and those whether in his or the other labs, or in management, that well didn't.

The Computer Science Lab (CSL) was a collection of engineers who weighed everything pitilessly against the question: How will this get us closer to our goal? They had commited themselves to developing Xerox's Office of the Future and anything that diverted their attention or served an alternative goal had to be discarded or obliterated.

It is Taylor's utterly single minded vision of interactive computing that drives much of the success and much of the drama of PARC. Taylor was in continual combat with the other labs for resources and funding and inevitably with his managers George Pake, Jerry Elkind and eventually Bill Spencer. But that was outside his lab. In it, he was the oil that kept the cogs turning and among his staff he was considered a unique and brilliant manager of researchers. It is on this skeleton of contradiction and conflict that the guts of the story of PARC hangs.

The book contains a number of particularly powerful scenes. Two particularly stuck out for me. The first is Alan Kay, his vision for a Personal Computer brusquely put down by CSL Manager Jerry Elkind, falling into a depression. Alan Kay is well known for his brilliance and verbal flourish but Hiltzik does well to also bring home his vulnerability in a way a modern reader would not expect. Kay would ultimately realise his vision of a Personal Computer - with the help of Taylor's CSL - while Elkind was seconded away from PARC on a Xerox taskforce. We tend to recall Kay's assertive (and largely proven) views on Computing. It is unexpected and moving then, particularly with the benefit of hindsight, to see him doubt himself and his ideas before they were fully realised.

The other scene involves Adele Goldberg, co-developer of Smalltalk, and her reactions to Apple's infamous raid on PARC. If you've ever seen The Pirates of Silicon Valley you might have a feel for how this all went down. But Hiltzik's account of it conveys such a sense of dread and hopeless frustration that the movie never came close to recreating.

By the end of the book Taylor's time at PARC draws to a close and with his departure so too does the most storied era of PARC. Scarcely six months after Taylor's forced resignation, the majority of his lab also resign and either follow him to Digital Equipment Corporation (creators of the famous PDP series of minicomputers), or join one of the many startups blooming in Silicon Valley following the success of IBM and Apple's Personal Computer products.

Dealers is fundamentally a story about people that just happen to be in technology - rather than a book about technology itself. It is a human story. It is about what happens when you take the cream of a generation's scientific talent put them in one place and throw lots of money at them. It is about what happens when you combine a visionary maverick with the proneness to credentialism by academically minded administrators. It is about what happens when you have corporate management that want to embrace change but either do not understand it, or worse, fear it.

It is the book's focus on the people of Xerox and PARC particularly, their feelings, motivations and backgrounds that brings this extraordinary tale of modern computing's birth to life.

Google Checkout: Error parsing XML; message from parser is: cvc-datatype-valid.1.2.1: '' is not a valid value for 'decimal'.

I have developed custom modules which add free or bonus items to a customer's cart if they use coupons, meet certain cart criteria or belong to particular customer groups. Buy x, get y rules also work this way. So this is a nuisance. Luckily few customers opt to use GoogleCheckout, but still, I don’t Live with Broken Windows[1].

Chasing the problem down the call stack leads to app/code/core/Mage/GoogleCheckout/Model/Api/Xml/Checkout.php and specifically the _getItemsXml() method.

$unitPrice = $item->getBaseCalculationPrice();
if (Mage::helper('weee')->includeInSubtotal()) {
    $unitPrice += $item->getBaseWeeeTaxAppliedAmount();
}
// ...
<unit-price currency="{$this->getCurrency()}">{$unitPrice}</unit-price>

Now, if the product's baseprice is 0, then for some unfathomable reason it's set to '', not 0. As the unit-price element expects a decimal value, an empty string fails validation.

The fix is pretty trivial

$unitPrice = $item->getBaseCalculationPrice();
if (Mage::helper('weee')->includeInSubtotal()) {
    $unitPrice += $item->getBaseWeeeTaxAppliedAmount();
}

$unitPrice = ((float) $unitPrice > 0) ? $unitPrice : 0.00;

The store I needed to fix only used US dollars so I haven't tested how the use of other currencies or locales might affect this fix.

To apply the fix, don't modify the core codepool, but instead take advantage of the local and community codepool's higher classloader priority[2] and place the amended code in app/code/local/Mage/GoogleCheckout/Model/Api/Xml/Checkout.php.

[1]: 'Don't Live With Broken Windows' is a tip I first read about in The Pragmatic Programmer. It is used to help fight Software Entropy (software's tendency to lose structure over time). This concept has parallels with the real world as urban areas with broken windows tend to see higher levels of vandalism when compared to areas where windows are constantly maintained.

When you ignore small problems it becomes easier to let more significant problems slide too. Hence the rule of thumb, 'Dont Live With Broken Windows'.

[2]: Magento resolves classes in this order local, community then core. This means if two classes have the name Mage_Core_Model_Foo one exists in local the other in core, then the version in local is used.

Thankfully there's a really simple way to socialise Nginx. It is by calling the nginx server command directly with the -s argument instead of using the /etc/init.d/nginx or service nginx commands.

-s lets you send signals to the Nginx master process and Nginx behaves differently whether it receives a quit signal versus a term signal.

# terminate the nginx master process immediately
$ sudo nginx -s stop 
# terminate the nginx master process once all outstanding connections have been completed
$ sudo nginx -s quit 

$ abonner@avalanche:~$ ps aux | grep nginx
root      1063  0.0  0.0  88796  3432 ?        Ss   Jan21   0:00 nginx: master process /usr/sbin/nginx
www-data  9786  1.3  0.0  91564  7352 ?        S    Feb03 190:11 nginx: worker process is shutting down
www-data  9788  1.3  0.0  91288  7072 ?        S    Feb03 189:02 nginx: worker process is shutting down
www-data  9789  1.3  0.0  91160  6956 ?        S    Feb03 190:03 nginx: worker process is shutting down

Let you visitors finish their drink, don't terminate nginx on a production server using /etc/init.d/nginx stop or service nginx stop.

Read more about Nginx's command line options

This year I didn't bother with any. I did in November though, resolve to read more technical books, and to particularly focus on the 'classics'. My motivation was stirred by Panagiotis Louridas's essay 'Rereading the Classics' from the book Beautiful Architecture. In it, Louridas examines the structure of Smalltalk and tries to reason why it achieved greater lasting success blazing a trail for others to follow than as a practical working programming environment.

Louridas suggests that Smalltalk is a classic and by way of justifying why, he quotes Italo Calvino's Why Read the Classics (1986)

The classics [...] exert a peculiar influence, both when they refuse to be eradicated from the mind and when they conceal themselves in the folds of memory, camouflaging themselves as the collective or individual unconscious.

A classic does not necessarily teach us anything we did not know before. In a classic we sometimes discover something we have always known (or thought we knew), but without knowing that this author said it first, or at least is associated with it in a special way. And this, too, is a surprise that gives much pleasure, such as we always gain from the discovery of an origin, a relationship, an affinity.

The classics are books which, upon reading, we find even fresher, more unexpected, and more marvelous than we had thought from hearing about them.

A classic is a book that comes before other classics; but anyone who has read the others first, and then reads this one, instantly recognizes its place in the family tree.

Smalltalk is a language that most of us have heard about but have rarely seen, and after being introduced to it I was inspired to start digging up any further literature I could find. The more that I read, the more that I appreciated its cleverness. Smalltalk can be eerily familiar, and as you begin to grok its syntax it is easy to recognise aspects that have inspired certain features of our 'modern' OO languages.

It is striking that the language itself (and indeed much of the material written about it) is so old. The most modern version of Smalltalk is Smalltalk-80 and although there are modern implementations, the overwhelming bulk of the syntax and environment still cohere to the 1980 standard. Yet download Pharo, read Alan Kay's The Early History of Smalltalk (1993), or Dan Ingalls's Design Principles behind Smalltalk (1981) and it all still seems contemporary. The language itself, which was controversial at the time for eschewing ALGOL syntax, has aged very well. Ruby has maybe made them fashionable, but Smalltalk sported things like block closures over a quarter of a century ago. Reflection, Metaprogramming and dynamic typing - all in Smalltalk in the 70s. Even the idea of using virtual machines to host the programming and operational environment seems remarkably contemporary today, as we increasingly move to abstract our programs and programming environments from bare metal.

It is humbling to see so many ideas that we take for granted today already implemented on a platform that is over a quarter of a century old. It is like amazing at how the Egyptians built the pyramids. Sure we could do it now, no sweat. But we have so much more raw engineering knowledge to throw at the problem. Alan Kay, Dan Ingalls, Adele Goldberg and the rest of the team at the Xerox PARC Learning Research Group designed and implemented Smalltalk with less computing power than my fridge has today. Despite formalising most of the vocabulary for OO software development during the development of Smalltalk it's hard not to feel like some of the energy and innovation of Kay's thinking didn't survive the C++ and Java succession.

Our discipline is still quite young when compared to traditional engineering or the sciences. Yet we seem to keep facing the same problems over and over again. The only difference is perhaps a few orders of abstraction, bigger piles of data and slightly more exotic technologies. But when I think that the fundamental concept of 'Agile' or at least 'iterative' development was doing the rounds in the 60s, it made me wonder what other insights are out there, buried in the forgotten past.

Maybe one reason we tend to forget what others have learned is that the average developer only reads one technical book per year. That means a sizeable percentage of software professionals do not actually read any! I think if I average over my professional working career (after graduating at the end of 2004) I would probably be batting one per year too. If I were to look over the last three years, maybe two per year. I wonder how many mistakes I might have avoided in that period had I read Fred Brooks's Mythical Man Month in 2004 rather than 2012.

My excuse has always been a lack of quality time. Last year in July I left my fulltime job for the world of consulting/freelancing. I imagined it would be easier to read more. In practice it has been, but not as easy as I thought or hoped.

I have a young daughter, my partner has returned to work and despite working from home, I still put in 50-60 hour weeks. It doesn't leave a lot of spare time and what spare time there is, is usually late at night when it's hard to concentrate.

So since I made a determined effort to read more, I've read four books cover to cover, and cherry picked bits out of another four. It feels good and the trick I've found is to read a chapter at a time, whenever you can. Whether it's just before dinner, over coffee/lunch, waiting at the supermarket or just before bed. I found that by reading, even if was just a little bit, every day, I was starting to get through entire books.

It requires conscious effort though, and some material is more suited to this style of reading than others. After a long day just before bed it's pointless trying to delve into SICP. It helps to read certain books during the work day, for example I've been re-reading GOOS and going through Kent Beck's TDD by Example over lunch times. I actively practice TDD so reading a chapter from one of these books midday helps me relate it directly to what I'm working on when I go back into the office.

I have a huge reading list setup in Google Reader, and I'm starting to think it's distracting. I have a slight OCD in that I need to keep the unread count at zero. At the end of a thirty minute work sprint, I would take five minutes to quickly flick through the list. It is distracting actually, and most of the content is superficial. The benefit of reading a book over a blog is the tendancy for a book to have its ideas more fully formed and logically structured. I realise the irony saying this while writing a blog myself. I feel that blogs have their place, but I have been spending far more time reading blogs than reading books. I am now leaving my Google Reader list unread longer and after a few 30 minute sprints picking up a book instead.

As I wrote above, Louridas inspired me to learn more about Smalltalk, and any research into the language leads to Alan Kay's ACM paper The Early History of Smalltalk. The paper beyond providing a wonderful insight into the language and the Xerox PARC is the source of some great quotes. One of my favourites goes:

Where Newton said he saw further by standing on the shoulders of giants, computer scientists all too often stand on each other’s toes.

In Software Engineering, we are so busy looking forward that we don't look back often enough. There is such a rich wealth of knowledge out there already considered and documented. I think we all should make more of an effort to re-discover it.

Here's a small selection I have picked up recently that I didn't know.

sudo !! 

Run the previous command as sudo. This is great when you realise you needed to run something as root.

ctrl-x e

Open up $EDITOR to enter in a long command. In my setup it fires up vim. This is great for some of the long rails commands you need to create controllers or models.

cat /etc/passwd | column -s':' -t 

Column, columnates input, the -t argument will format standard input into a table and -s lets you specify an arbitrary field delimiter. For unformatted input this is very handy.

These next few are specific to zsh, and while I do love bash, since switching to zsh I haven't really looked back. It's things like this that when you work with a terminal every single day, you can't give up.

aaron@tempest ~ $ d                                 
0	~
aaron@tempest ~ $ cd /etc
aaron@tempest /etc $ d
0	/etc
1	~
aaron@tempest /etc $ 1
~
aaron@tempest ~ $  

The 'd' command lists the directory stack, and then entering an integer will switch you directly to the directory index in the stack. It is a killer app.

Moving directories also is very pleasant in zsh. Use '..' to move up a directory, and simply type the name of the directory in, to move into a directory.

aaron@tempest ~ $ ..       
aaron@tempest /Users $ aaron
aaron@tempest ~ $ 

This last one is a trick I've know for a few years, I don't know how much time this has saved me exactly, but I use it every single day.

In vim, if you're editing a file that requires root (or any other user) permissions, you can write the file by doing

:w !sudo tee %

I use it so much that I've set up a leader key binding in my .vimrc

nnoremap <leader>sr :w !sudo tee %<CR>

There's nothing more annoying than making lengthy changes to a config file, go to write it and getting permission denied...

I make all my configs available online at github, if you're interested in seeing how I setup my environment.

To cut a short story, very short, you do this (at least if you have macports in /opt/local (the default), and are using the mysql55 package).

$ gem install mysql2 -- --with-mysql-lib=/opt/local/lib/mysql55/mysql --with-mysql-include=/opt/local/include/mysql55/mysql 

This matters a lot when you want to use xdebug on a project that runs off a virtual machine, or even perhaps a remote server.

The typical pattern (when using PHPStorm in my case) to invoke a remote cli debugging session is to set an environment variable telling the IDE what server configuration to use, and to tell PHP what remote host to connect to.

$ PHP_IDE_CONFIG='serverName=mydevmachine.local' php -dxdebug.remote_host=192.168.0.1 myphpscript.php

Now this will work fine, however if we want to debug during a phpunit test normally you would do this

$ PHP_IDE_CONFIG='serverName=mydevmachine.local' phpunit -dxdebug.remote_host=192.168.0.1 -c phpunit.xml

Unfortunately this doesn't appear to work at the moment (version 3.7.9). If I use the xdebug test client, I can see xdebug trying to connect to the localhost, ignoring what I've told PHPUnit. I'll look into this a bit more later, but I suspect PHPUnit isn't passing on the php.ini settings in a timely fashion for xdebug to hook into.

The solution to this problem is to make use of ssh port forwarding. This works exactly the same for a virtual machine as it would for a remote host, which makes xdebugging on a production machine (hopefully only ever in an emergency!!!) much more straight forward (and less insecure).

$ ssh -R 9000:localhost:9000 myvm.local

This sets up myvm.local to forward all connections to its localhost on port 9000 to the remote client's port 9000. When xdebug goes to connect to localhost:9000, it ends up actually connecting to mydevmachine.local:9000.

It's a bit of a hack, but a time saving one. The other alternative is Vim and its xdebug plugin. This isn't a bad alternative. But once you've experienced the power of PHPStorm's debugging implementation it's hard to go back.

It has been an exciting time to be a PHP Developer these past twelve months, PHP 5.3 is now rock solid and PHP 5.4 is getting there. Both releases significantly modernise elements of the language, closing the gap between PHP and the offerings of more 'in vogue' languages.

In technology we often see change happen in sudden, explosive steps. Often it seems to coincide with developments a technology's ecosystem or among its competitors. For PHP the first major kick was the rapid rise in popularity of Object Oriented Programming in the early 00s. This led to PHP 5's radically overhauled OO implementation in 2004. The next kick, I feel, came in 2005 when Ruby on Rails exploded into everyone's consciousness. RoR provided a full stack web development platform that drastically simplified creating complex web applications. The PHP community responded in kind with a number of 'fullfat' Model View Controller (MVC) Frameworks, the most successful being Zend's and Symfony.

It all depends

The arrival of PHP 5.3 and features like Namespaces, PHAR, Closures and the ubiquity of Github is having the effect of giving PHP a new kick, and the results are starting to make themselves felt. We now have second generation frameworks from Zend and Symfony leveraging these technologies.

One problem remains though, and that is managing and distributing dependencies. Modern web development platforms all now have robust dependency management tools available and in the PHP camp, PEAR wasn't really cutting it.

The success of Symfony2 in particular, with its emphasis on high quality, modular components, forced PHP developers to address how they bundled and distributed library code.

Luckily for us, the guys behind Composer, (again, taking considerable cues from the Ruby community) have licked it. Composer, in tandem with Symfony2 components allow PHP Developers to confidently build on top of other developers' libraries.

Do we really need another packaging tool?

Why did we need another package and dependency management tool anyway? What really, is wrong with PEAR? Well, if we wind the clock way back to 1999 when Netscape Communicator was still the most popular web browser and Google had just moved out of Susan Wojcicki's garage, PEAR was conceived as PHP's answer to PERL's CPAN. Despite some strident efforts, it never really managed to become the most pleasant package manager to work with: rigid, elitist and worst of all, difficult for end-users. PEAR's age strictly speaking is not the problem, but its centralised nature is a bottleneck and there is no straightforward way to handle two packages with varying dependencies. For example: say package x, requires stable package y. Package z requires beta package y. You can't install both. Dependency and package management has moved on a long way since 1999.

Over time PEAR's shortcomings have led to a graveyard of abandoned packages, code of at best variable and at worst, dubious, quality, and a community lacking in any sort of dynamism. If you make something easy, people will use it. PEAR is difficult to use for developers and users alike.

Composer democratises (in the best sense) things and puts full control of dependencies in the hands of library developers. Free to pick and chose code they want to use. Free from having to worry about navigating the PEAR jungle. Here the rise and rise of Github has been key. Composer can sit over the top of code distribution services provided by Github, or it can use its default Packagist repository. This removes the need for libraries to live in a blessed canonical repository or for developers to host it themselves.

... profit?

There's no compelling need now to constantly rewrite basic library components (I think we've finally licked what ought to be the basic issue of class loading!), Free of the shackles of PEAR, we are witnessing an explosion of high quality PHP frameworks, libraries and utilities.

PHPSpec, Behat, Twig, Mockery, Doctrine, are just a few that immediately spring to mind. Some (such as Doctrine) have been around a while. However the advances PHP 5.3 brought to the table have significantly helped improve the utility of these projects.

Anyway, so (after a fashion) I come to the tool that motivated me to write this post, n98-magerun.

The name is horrible, but the tool itself is brilliant. In short, it's Drush for Magento and it's wonderful. It is one of those tools that makes you wonder what on earth you did before it.

I have a folder full of bash scripts, cobbled together to help automate the mind-numbing process of managing Magento installations. Over the course of a few months Christian Münch and friends have overseen a small tool quickly develop into the kind of utility we've all wanted but never had the time/patience to build ourselves.

Magerun is elegantly simple for the user, and cleanly extendable by developers. It is a perfect illustration of why it's such a great time to be a PHP developer. Better dependency management, easy distribution, modular libraries and powerful language syntax have all came together to let someone with an itch, scratch it quickly and effectively.

It has become several orders of magnitude easier to develop, package and distribute PHP libraries and utilities. The result of this leap forward is a brilliant tool that helps Magento developers dramatically increase their productivity.

I'll be writing about how awesome Magerun is shortly, but just one of its cool features is the ability to dump out a merged version of Magento's config.

This is extremely helpful when trying to resolve conflicts between modules, or figure out what bit of configuration is taking precedence.

The resulting xml though is pretty raw and unformatted, but xmllint can fix that.

Xmllint expects a file to work with and cannot use piped input. So we need to use bash's Process Substitution feature to avoid having to create temporary files.

$ xmllint --format <(magerun config:dump)

So, magerun and xmllint, a simple way to get a formatted, easy to examine view of how Magento is putting your install's configuration together.

$ git show --name-only <commit>

This will give you a list of affected files in commit

Alternatively if you don't care what differs in the specific contents between two branches, and only want to see different files you can do

$ git diff master..origin/master --name-only

This will show you the list of files that are different between your local master branch and the remote master branch. Handy if you have just done a git fetch and want to see what's different before merging or rebasing.

First thing, shut down the running instance and then restart it directly

$ sudo -u <mysql_user> mysqld_safe --skip-grant-tables --skip-networking

The --skip-networking option is important, as by skipping the grant tables, any user can connect to the running mysqld service, will full permissions.

Once you've started the server up, login without a password, and issue an update query to the mysql.user table.

$ mysql -uroot mysql
mysql> UPDATE user SET password=password('newpassword') WHERE User = 'root'

Close down mysqld and restart. You're good to go.

We've seen some pretty epic PHP rants this year, probably the most famous among them are PHP a Fractal of Bad Design, and Jeff Atwood's latest (in what seems to be a biennial broadside) The PHP Singularity.

The common thread in these rants is incredulity that anyone would, in 2012, write new code in PHP. There's a lot of reasons why someone might write greenfield PHP code in 2012. But equally (and this is said as a decade long PHP programmer) I have to admit that plenty of the criticisms levelled at PHP are valid. Yet, for the most part they just don't, particularly, matter.

One criticism that is wholly invalid yet comes up time and time again, is that using PHP intrinsically leads to bad code. I don't feel this is an inherent trait of PHP itself, so much symptomatic of the popularity and low barrier to entry of PHP. Basically there's more examples of bad code out there compared to pretty much any other platform because there's simply more code out there, written by programmers of wildly varying skill. The other problem, is that PHP came into existence as a scratch to a C programmer's itch. PHP was developed at a time where people still actually wrote web applications in C and where the stateless nature of HTTP was relatively respected. PHP, like the web itself, has moved on dramatically since then.

A modern PHP 5.4 webapp looks about as similar to an early 00s PHP 4 webapp as Scala does to Java. Yet many critics when slating the language appear to be code archeologists, excavating pre-historic practices that went out of favour long-ago.

This can be partially forgiven, because owing to the age of the language there's plenty of out of date information out there with high rankings in Google. The ubiquitous w3schools is an unfortunate example of bad practices coming well ahead of sites with more modern approaches to solving problems in PHP.

So 'New PHP' is very different to 'Old PHP'. But the 'Old PHP' is what most people seem to find when searching in Google and this confuses people.

We see this manifested in blogs like Will Rails become the new PHP. This blog has a number of spectacular shortcomings, most egregiously the author's horribly naive view of the PHP community, but the interesting one is his ignorance of the power of Google.

There's plenty of support out there for budding PHP programmers, whether on the web, on forums, or IRC. There are countless, well attended, supported and growing conferences, meetups and the like for PHP programmers. One thing that a community cannot do, is force Google to nuke w3schools' PageRank. Which means that brilliant efforts like PHP The Right Way get swamped by old, incorrect and at times dangerous dreck.

And what the 'Will Rails become the new PHP' author perhaps hasn't realised, is Rails, at least in the terms he's trying to couch it, has already become the 'New PHP'. I am a novice Rails developer, I like to hack around in it as it can be quite fun to spike out solutions. What I've been struck by, is the sheer amount of bad advice out there. Advice novices will come across, if they turn to Google for help.

If you're well versed in a platform you learn through brutal experience what works, what doesn't and your nose is finely tuned to bullshit. When I read a PHP article I know instinctively if what I am reading is reliable. But with Rails, as a novice, I don't quite have that sense, beyond my own background experience with other programming languages.

So let's look at an example. I've been working on a dead simple Rails authentication webservice. It listens for HTTP requests for /login, /logout, /session, etc., and emits either XML or JSON in response. I'm using the respond_to method to serve out these responses. Unfortunately what I found is if I request a route that does not exist, I get an HTML error back. This doesn't make a lot of sense for a webservice that otherwise speaks XML and JSON.

Other global exceptions similarly respond with HTML. I don't want to wrap every action up in a begin/rescue block and there is certainly no way to intercept router exceptions in actions anyway. So I needed to learn how to catch global exceptions.

In my journey of (Google) discovery I came across this blog post and appeared to hit paydirt. The advice appears to be legit, hell someone in the rails community even featured it in a podcast. So on the face of it this seems good. But that switch statement sure is smelly. Does it, really, need to be this hard?

Now my general purpose programming brain recognised that this code while solving my problem, is not ideal. And why is it not ideal? This one method sure has a lot of responsibility. Method names with plurals in them are usually a code smell. Over time as more specific exceptions need to be handled I would end up with code that is as easy to read as Goethe's Faust, photocopied and in the original Gothic script (not easy). What we have here is a God Method in training.

Now, what if I am a Ruby/Rails/Programming novice? I have got plenty of other stuff to learn, I'm going to go right here and Cargo Cult this code into my webapp and move on. Just like all the rookie PHP coders do, right?

Well, I didn't do that. I saw that this rescue_from method was pretty awesome and so I went to the Rails API docs to look it up.

API docs for any language are pretty terse, but what jumped out at me was this line:

“Handlers are inherited. They are searched from right to left, from bottom to top, and up the hierarchy. The handler of the first class for which exception.is_a?(klass) holds true is the one invoked, if any.”

This isn't great documentation admittedly, but it means basically, if you put rescue_from Exception at the bottom of the list of rescue_from handlers in your application_controller.rb file, then since everything derives from Exception, nothing else will get a look in (Rails will look at the handlers from the bottom up).The author of that helpful blog we found didn't realise this, and so his solution was needlessly complicated.

What can we learn from this? Well Rails programmers certainly live in a glass house and shouldn't throw stones is one thing. But on a slightly less trollish note, there is a problem here for all novice programmers that turn to Google to help them solve problems. The answers on Google are usually either wrong, or at best, incomplete. As the web gets older, bad and out of date advice piles up making it much harder for novices to find good advice.

Knocking a language for this phenomenon (or a framework, seriously, whatever) is more than a little ignorant and doesn't solve the problem. Efforts like PHP The Right Way is how PHP is trying to fix it. If Rails really doesn't want to be the 'Old PHP', they need to realise it's less to do with languages and platforms, and more about SEO.

To see what branches your local repo things exists you do something like this:

$ git branch -rv
> origin/1620-upgrade  2e0cc56 Ignore active local.xml from vc
> origin/HEAD          -> origin/master
> origin/cas-sso       2351be5 Add gateway logiin and logout support
> origin/giveaways   63daf5a Use cms blocks for banner placements
> origin/master        496c975 Merge affiliate module
> origin/newskin      d7220c9 Optimise skin and ui images
> origin/release       496c975 Merge affiliate module

So this is my local Magento git repository, many of the branches here are now defunct and no longer in the remote (i.e. I had previously had used $ git push origin :branch from another host)

To refresh then I need to prune my branches list. The git incantation to do this is

$ git remote prune origin
> Pruning origin
> URL: dev@vcs:git/store.git
* [pruned] origin/1620-upgrade
* [pruned] origin/giveaways
* [pruned] origin/newskin

Looking at the remote branch list again:

$ git branch -rv
> origin/HEAD    -> origin/master
> origin/cas-sso 2351be5 Add gateway logiin and logout support
> origin/master  496c975 Merge affiliate module
> origin/release 496c975 Merge affiliate module

A simple configuration recipe for adding new cache tags to the Magento backend’s “clear cache” feature.

In theory the Magento cron subsystem should keep a lid on these tables growing too big, but I avoid using Magento cron, preferring to handle that myself directly via crontab tasks.

The other option is to write your own table cleaning script (or copy one from somewhere), and this will work too. But it's annoying, if you don't want this log data, why write it in the first place.

So my solution is to disable it by removing the observer events that perform the logging.

I have this in my local.xml which takes precedence over other nodes in the config and therefore overwrites them. Here, by setting the observer to be the string 'disabled', the existing observer event is removed and replaced with something that will never be fired.

Now, you don't need to worry about periodically cleaning out your database, nor do you need to fear a 3am text message from your production DB servers screaming about the disk being full...

I'm going through some PCI scan results this morning, and in the main it's going well, but I got a couple XSS hits on our catalogsearch pages. This is odd, I think. I've audited these pages, they definitely get routed through magento's escaping code.

On closer examination it turned out the form was okay, it was via the breadcrumbs, that unescaped input was getting into the wild.

I'm running Mage 1.6.x so this code may look a little different if you're running 1.7

Take a look at app/code/core/Mage/CatalogSearch/Block/Result.php, and specifically at the prepareLayout() method:

Now if you look at line 11, if breadcrumbs are enabled, unescaped input is happily added ready for output.

$title = $this->__("Search results for: '%s'", $this->helper('catalogsearch')->getQueryText());

This fix is easy, replace line 10 with:

$title = $this->__("Search results for: '%s'", $this->helper('catalogsearch')->getEscapedQueryText());

This is a really neat example of the evils of duplication and where bad programming practice can lead to real world problems. I am speculating, but it seems reasonable to infer that the original programmer got trigger happy with the copy & paste keys. Later, at some point you could imagine another engineer coming in to XSS safe the code fixed one bit, but (and programmers are human) missed the other (exactly the same line), and we end up with an issue like this.

Personally, I patched the file as described above and stuck it in app/code/local/Mage to override the core code pool version.

Unfortunately this doesn't happen very often and when using a stack of Rubygems, you almost always get bitten by something.

My cause for complaint today is Vagrant and Chef, well specifically Chef Solo. Vagrant is fine, it does what you tell it to do, but for most use-cases Chef Solo is the right tool to use for provisioning your virtual server. The Vagrant docs on Chef Solo unfortunately fib you, they say you can use Data Bags with Chef Solo, but by default you cannot.

This is a big deal as many useful Chef recipes make heavy use of Data Bags. Data Bags which let you provide environment specific configuration for your provisioning is not yet supported by the stock Chef Gem (currently version 10.12.0). In order to make use of Data Bags with Chef Solo, you need version 10.14.0 and above. This means building the gem from source.

I use Veewee to build my vagrant base boxes (you should too, it's awesome!), and you can edit the postinstall.sh file in your box definition folder to build Chef from source, rather than installing it directly via Rubygems.

You can repeat this for your local dev machine, and now you can get Chef Solo cooking up your recipes and happily using data bags.

This overrides the core code pool's adminhtml definition, and puts a dependency on a non-existent module. Effectively, this disabled the menu item because it no longer meets the defined dependency requirements.

As always with any magento configuration / module changes, you may need to clear caches for this to take effect.

Yesterday I had to do an emergency restart of mysql in production and the resulting magento report/xxxx files swamped out everything else that I might have wanted to look at.

So specifically, I wanted to delete all the files that were created between a start and end date.

GNU find makes this easy

$ find . -type f -newer 655958024588 ! -newer 627812220533 -ls

This instructs find to list (-ls) all files (-type f) that are newer than a file called 655958024588 (-newer) and not newer than 627812220533 (! -newer).

If you do not have two files to act as date range boundaries, you can use touch to create them.

$ touch -t yyyymmddHHMM start_date_file
$ touch -t yyyymmddHHMM end_date_file

Then supply these file names to -newer and ! -newer.

To delete the files we can use -exec.

$ find . -type f -newer 655958024588 ! -newer 627812220533 -exec rm -rf {} \;

Here it's the -exec argument does the heavy lifting. {} is a placeholder for the file name (find substitutes '{}' with each found filename) and ; terminates the command sequence (much like it does in regular bash).

So, if you want to use veewee to setup a new amd64 Precise Pangolin basebox for vagrant, you either have to pull the latest veewee sources from github, or download the most recent releases templates and copy them over into your veewee gem folder.

This is where veewee-templates-update steps in, it automates that latter step (downloading and installing just the updated templates) for you.

Installation is simple:

$ gem install veewee-templates-update

Then, just run the updater:

$ veewee-templates-update
> Veewee: /home/aaron/.rvm/gems/ruby-1.9.3-p194/gems/veewee-0.2.3
> Downloading: https://github.com/jedi4ever/veewee/tarball/master
> Extracting: CentOS-4.8-i386 CentOS-5.5-i386-netboot CentOS-5.5-x86_64-netboot 
> ...

$ vagrant basebox define precise-amd64 ubuntu-12.04-server-amd64
> ...
> (profit)

Anyway, quite happy with it so far, but like any geek with new kit, I want to know everything about it, and make it dance to my whim.

Two things learned this morning: if you have an old Magsafe (1) powerpack, which I did from my old 13" Mac Pro, and it has higher or equally rated wattage you can use it with your macbook air.

It makes sense, a high rated powersupply can support a lower power rated device, but not vice versa. So a 60 watt Macbook Pro magsafe can power a 45 watt Macbook Air. But a 60 Watt Magsafe can't power a 15" 85 watt macbook pro.

The other thing which I have discovered, is that the thunderbolt port on the newer Airs use the same socket form factor as the mini display port. This means is if you have an old Mini DisplayPort -> DVI adapter lying around, you can reuse it.

My first real job has been to get a decent unix environment up. OSX's BSD utilities don't really cut it. Macports is far and away the best distribution out there.

Once you install coreutils and get ls, find etc it makes sense you will want to change your shell to a modern version of bash (or zsh if that's the way you roll).

Lion ships with bash 3.2 whereas Macports will give you a contemporary version 4.2 Unfortunately it's not as simple as going

$ sudo port install bash
$ chsh 
  <input /opt/local/bin/bash>

I had to do this before, but I'd forgotten there's a trick to changing your shell in OSX to a non-standard location. The file /etc/shells contains a list of valid shells chsh will permit. You need to edit (as root or via sudo) this file and add your macports shells. Once that's done chsh will let you change no problem.

To take a column from a MySQL database and quickly output it ready formatted as a Javascript array literal (without any specific escaping) do:

echo 'SELECT column FROM table WHERE some_column = "somevalue"' | mysql -uuser -ppass --silent yourdb | awk -v q="'" '{ print q $0 q }' | paste -s -d ',' | sed 's/(.*)/[\1];/'

The first part of the command is self explanatory, you pipe in a query to mysql, and ask it to give you raw unadorned output. It will return each row for column 'column' from table 'table' as a line of output.

You pipe it to awk and ask it to wrap the values in single quotes. Due to shell escaping with single quotes, you set the q variable to a single quote. Paste then joins all the output lines together separated by commas.

Finally I use sed to wrap the resulting output in Javascript array literal '[' and ']' symbols. Awk or any other tool to concatenation approach would do just fine here too.

field_name: [Start TO Finish] 

You can also use wildcards and specific constants in a logical way e.g:

[NOW TO *] or [* TO *]

To search over documents that do not have a value for that date field, e.g. is NULL, you use the syntax:

-field_name: [* TO *].

It is hard though, to search for dates that are EITHER NULL OR lie in a specific range.

It would seem logical to specify

date_field:[Start To Finish] OR -date_field: [* TO *] 

Unfortunately Solr does not appear to support specifying a field multiple times in this way.

So the trick is to effectively query for everything you do not want, then negate the result.

This approach says select me anything that isn't in the range of this date and is not null. When you invert that result, you get all the documents sit inside the date range or are NULL.

The query to weave this magic is

-(-date_field:[Start TO Finish] AND date_field:[* TO *])

A Singleton, if you've not heard the term before, was popularised in the Design Patterns book by the Gang of Four (Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides). To be very succinct, a Singleton is a way to ensure there is only ever one instance of a class in an Object Oriented design. To put it in even simpler terms , it is an Object Oriented version of a global variable.

It's used heavily in Magento (in the app/code/core directory, 2261 times in fact!). But anyway, why is it considered harmful? There are a number of arguments for why and why not. Herb Sutter's Once is not enough gives a pretty good (and fun to read) overview of them, or you can read Kenton Varda who looks in-depth at the topic. I generally think though, that in Object Oriented software, you're seeking to create abstractions around complexity. The Singleton is a (too) convenient escape hatch from encapsulation and can lead to the attendant issues you get with global variables.

In the Magento/PHP land a more implementation specific problem with Singletons, is memory consumption. Today I was revisiting a Magento Promotions extension I had written and trying to figure out why it was suddenly obliterating PHP's memory_limit.

This extension basically piggybacks on the existing Promotions/Coupons system, but generates an index of products that match coupon codes, the price before and after the promotion is applied and some other metadata.

In order to determine what products have a promotion associated with them, I run through all the products, and match SalesRule conditions against them. I create a synthetic quote for the products that match and then pump them through the SalesRule validator. This effectively applies the promotion to the product and let us see what the savings are.

It's fairly basic, it doesn't look at multi product combinations but it works well enough for simple cases.

/**
 * Apply pricing rules to a synthetic quote to calculate discounted price
 * 
 * @param string $couponCode
 * @param Mage_Catalog_Model_Product $product
 * @return  float
 */
public function applyToProduct($couponCode, $product)
{
    $quote = Mage::getModel('sales/quote');
    $item = Mage::getModel('sales/quote_item')
        ->setQuote($quote)
        ->setProduct($product)
        ->setQty(1)
        ->setBaseDiscountCalculationPrice($product->getPrice())
        ->setDiscountCalculationPrice($product->getPrice());

    $validator = Mage::getSingleton('salesrule/validator')
        ->init(1, 1, $couponCode);
    
    $validator->process($item);
    
    return $product->getPrice() - $item->getDiscountAmount();
}

Now when I wrote this code, it seemed sensible to use the validator as a Singleton, after all I only needed one copy of it. It didn't, at the time, seem to make sense to create and then destroy the validator a couple of thousand times during indexing. Indeed when this code was first deployed, everything ran smoothly.

Recently the user of this extension added a whole bunch of sales rules - and this caused that product/salesrule index loop to detonate.

That Singleton Validator, which was written as some sort of optimization, started happily hosing over a gig of ram.

Changing getSingleton() to getModel() took ram usage down from 1100MB to about 80MB.

My suspicion is that PHPs garbage collection wasn't cleaning up adequately after each validation attempt. As the validator is effectively static, it never gives up its references for PHP to clean up. When you use getModel(), the validator loses all its references after each loop. While it means it also has to be constructed after each loop, but that allows PHP to free the memory it is using.

The Singleton is already a controversial pattern these days, but Magento developers should be particularly wary of it's implementation and its scope to hose memory.

Typically I manage PHPStorm by extracting it to /opt and then symlinking the extracted folder to /opt/PhpStorm.

To create a nice launcher for Unity you create a desktop entry under ~/.local/share/applications

$ vim ~/.local/share/applications/jetbrains-phpstorm.desktop

Now paste the following in (adjusted for your own paths)

[Desktop Entry]                                                                  
Version=4.0.1                                                                   
Type=Application                                                                
Name=JetBrains PhpStorm                                                         
Exec=/opt/PhpStorm/bin/phpstorm.sh %f                                           
Icon=/opt/PhpStorm/bin/webide.png                                               
Comment=Develop with pleasure!                                                  
Categories=Development;IDE;                                                     
Terminal=false                                                                  
StartupNotify=true                                                              
StartupWMClass=jetbrains-phpstorm           

Hit your windows or super key and type in 'Php' - you should see the newly created desktop entry there. Once launched and fully started, you can opt to keep PHPStorm locked to the Unity launcher for easy startup.

Even today, almost all x86 linux distributions can run (in theory) on a 32bit 386 processor. Let's be clear though, while it is a remarkable (and successful) processor, the 386 is an antique. The selling point of Gentoo was that it was one of the few distributions to give you the power to build a system, specifically for modern processors abandoning backwards compatibility for long-obsolete ones.

In the early days you had to start from what was called a (and I don't think they do it this way anymore) 'stage 1' install. A stage 1 install is where all you have is a bootable livecd with a few basic set of tools: a c compiler, a shell and the basic gnu coreutils...and irssi. Enough utilities to allow you to build further C packages. You'd run the command 'emerge system' and it would go off and build gcc, glibc, coreutils etc. Once these were built you'd then rebuild glibc, and gcc with your newly compiled, architecture specific, compiler and c library.

Anyway, this took a long time, particularly on sub gigahertz Pentium 2s and 3s, and Gentoo systems tended to break a lot, and by break, in the absolute best case, I mean merely became un-bootable.

The process to recover the system was pretty much the same as to install it, you had to boot from a livecd, configure your network card, hook up to the network, then chroot to the broken disk. At this point you could try and repair whatever damage you had caused.

These days if you do something silly, like I don't know, try and dist-upgrade from Ubuntu Oneiric to Precise, you can get that true Gentoo feeling (i.e. nothing works and you can't boot the machine).

This happened to me this afternoon and the hard yards done with Gentoo came to the rescue.

Here's how you do it.

Boot up from a livecd (or usbkey), get the network card modules loaded and get a dhcp address. With the Precise Live DVD you can do all of this pretty easily by selecting the 'try without installing' option from the bootloader.

Once you're online, you need to prepare the mount. First step is to mount the root partition somewhere, I typically use /mnt/ so say /mnt/oneiric (it can be whatever). If you're not sure of your partition numbers, your livecd will almost certainly come with fdisk with which you can push 'p' to print the partition table.

$ mount -t ext4 /dev/sda5 /mnt/ubuntu

If you have a separate boot partition mount that too.

$ mount -t ext2 /dev/sda1 /mnt/ubuntu/boot

Now in order to have a functional chroot, we need the proc, dev and sys subsystems to be mounted onto the chroot. This is the tricky bit.

$ mount -t proc none /mnt/ubuntu/proc
$ mount -o bind /dev /mnt/ubuntu/dev
$ mount -o bind /sys /mnt/ubuntu/sys

In the case of the sys and dev dirs, we need to reference the exact same mountpoints as the host so we use the -o bind option.

Last thing, we want to have functional network name resolution so we copy over the host's /etc/resolv.conf to /mnt/ubuntu/etc/resolv.conf

Now the chroot is ready

$ chroot /mnt/ubuntu /bin/bash
$ source /etc/profile

The chroot will be pretty much as it would be if you'd booted into it normally with a few exceptions. The kernel and kernel modules will be those of the host. If you need to access some specific hardware you need to set this up on the host.

My busted Precise install was fixed with a simple apt-get update and upgrade, before re-running the grub installer.