/dev/notes

It is 2013, we (still) don't have flying cars, or hoverboards, AND as developers, we still use terminals to interact with our operating system. So every so often I like to browse through commandlinefu.com and try and pick up any little tidbits which improve my command line efficiency.

Here's a small selection I have picked up recently that I didn't know.

sudo !! 
  

Run the previous command as sudo. This is great when you realise you needed to run something as root.

ctrl-x e
  

Open up $EDITOR to enter in a long command. In my setup it fires up vim. This is great for some of the long rails commands you need to create controllers or models.

cat /etc/passwd | column -s':' -t 
  

Column, columnates input, the -t argument will format standard input into a table and -s lets you specify an arbitrary field delimiter. For unformatted input this is very handy.

These next few are specific to zsh, and while I do love bash, since switching to zsh I haven't really looked back. It's things like this that when you work with a terminal every single day, you can't give up.

aaron@tempest ~ $ d                                 
  0    ~
  aaron@tempest ~ $ cd /etc
  aaron@tempest /etc $ d
  0    /etc
  1    ~
  aaron@tempest /etc $ 1
  ~
  aaron@tempest ~ $  
  

The 'd' command lists the directory stack, and then entering an integer will switch you directly to the directory index in the stack. It is a killer app.

Moving directories also is very pleasant in zsh. Use '..' to move up a directory, and simply type the name of the directory in, to move into a directory.

aaron@tempest ~ $ ..       
  aaron@tempest /Users $ aaron
  aaron@tempest ~ $ 
  

This last one is a trick I've know for a few years, I don't know how much time this has saved me exactly, but I use it every single day.

In vim, if you're editing a file that requires root (or any other user) permissions, you can write the file by doing

:w !sudo tee %
  

I use it so much that I've set up a leader key binding in my .vimrc

nnoremap <leader>sr :w !sudo tee %<CR>
  

There's nothing more annoying than making lengthy changes to a config file, go to write it and getting permission denied...

I make all my configs available online at github, if you're interested in seeing how I setup my environment.

GNU Find never ceases to amaze me with its utility.

Yesterday I had to do an emergency restart of mysql in production and the resulting magento report/xxxx files swamped out everything else that I might have wanted to look at.

So specifically, I wanted to delete all the files that were created between a start and end date.

GNU find makes this easy

$ find . -type f -newer 655958024588 ! -newer 627812220533 -ls
  

This instructs find to list (-ls) all files (-type f) that are newer than a file called 655958024588 (-newer) and not newer than 627812220533 (! -newer).

If you do not have two files to act as date range boundaries, you can use touch to create them.

$ touch -t yyyymmddHHMM start_date_file
  $ touch -t yyyymmddHHMM end_date_file
  

Then supply these file names to -newer and ! -newer.

To delete the files we can use -exec.

$ find . -type f -newer 655958024588 ! -newer 627812220533 -exec rm -rf {} \;
  

Here it's the -exec argument does the heavy lifting. {} is a placeholder for the file name (find substitutes '{}' with each found filename) and \; terminates the command sequence (much like it does in regular bash).

A quick bit of shell-fu.

To take a column from a MySQL database and quickly output it ready formatted as a Javascript array literal (without any specific escaping) do:

echo 'SELECT column FROM table WHERE some_column = "somevalue"' | mysql -uuser -ppass --silent yourdb | awk -v q="'" '{ print q $0 q }' | paste -s -d ',' | sed 's/(.*)/[\1];/'

The first part of the command is self explanatory, you pipe in a query to mysql, and ask it to give you raw unadorned output. It will return each row for column 'column' from table 'table' as a line of output.

You pipe it to awk and ask it to wrap the values in single quotes. Due to shell escaping with single quotes, you set the q variable to a single quote. Paste then joins all the output lines together separated by commas.

Finally I use sed to wrap the resulting output in Javascript array literal '[' and ']' symbols. Awk or any other tool to concatenation approach would do just fine here too.

For about eight years I ran Gentoo Linux before I eventually gave it up, and moved on to Ubuntu. It was remarkable in that it provided a BSD like ports system and let you compile your system from the ground up. It also tended to break, a lot.

Even today, almost all x86 linux distributions can run (in theory) on a 32bit 386 processor. Let's be clear though, while it is a remarkable (and successful) processor, the 386 is an antique. The selling point of Gentoo was that it was one of the few distributions to give you the power to build a system, specifically for modern processors abandoning backwards compatibility for long-obsolete ones.

In the early days you had to start from what was called a (and I don't think they do it this way anymore) 'stage 1' install. A stage 1 install is where all you have is a bootable livecd with a few basic set of tools: a c compiler, a shell and the basic gnu coreutils...and irssi. Enough utilities to allow you to build further C packages. You'd run the command 'emerge system' and it would go off and build gcc, glibc, coreutils etc. Once these were built you'd then rebuild glibc, and gcc with your newly compiled, architecture specific, compiler and c library.

Anyway, this took a long time, particularly on sub gigahertz Pentium 2s and 3s, and Gentoo systems tended to break a lot, and by break, in the absolute best case, I mean merely became un-bootable.

The process to recover the system was pretty much the same as to install it, you had to boot from a livecd, configure your network card, hook up to the network, then chroot to the broken disk. At this point you could try and repair whatever damage you had caused.

These days if you do something silly, like I don't know, try and dist-upgrade from Ubuntu Oneiric to Precise, you can get that true Gentoo feeling (i.e. nothing works and you can't boot the machine).

This happened to me this afternoon and the hard yards done with Gentoo came to the rescue.

Here's how you do it.

Boot up from a livecd (or usbkey), get the network card modules loaded and get a dhcp address. With the Precise Live DVD you can do all of this pretty easily by selecting the 'try without installing' option from the bootloader.

Once you're online, you need to prepare the mount. First step is to mount the root partition somewhere, I typically use /mnt/ so say /mnt/oneiric (it can be whatever). If you're not sure of your partition numbers, your livecd will almost certainly come with fdisk with which you can push 'p' to print the partition table.

$ mount -t ext4 /dev/sda5 /mnt/ubuntu
  

If you have a separate boot partition mount that too.

$ mount -t ext2 /dev/sda1 /mnt/ubuntu/boot
  

Now in order to have a functional chroot, we need the proc, dev and sys subsystems to be mounted onto the chroot. This is the tricky bit.

$ mount -t proc none /mnt/ubuntu/proc
  $ mount -o bind /dev /mnt/ubuntu/dev
  $ mount -o bind /sys /mnt/ubuntu/sys
  

In the case of the sys and dev dirs, we need to reference the exact same mountpoints as the host so we use the -o bind option.

Last thing, we want to have functional network name resolution so we copy over the host's /etc/resolv.conf to /mnt/ubuntu/etc/resolv.conf

Now the chroot is ready

$ chroot /mnt/ubuntu /bin/bash
  $ source /etc/profile
  

The chroot will be pretty much as it would be if you'd booted into it normally with a few exceptions. The kernel and kernel modules will be those of the host. If you need to access some specific hardware you need to set this up on the host.

My busted Precise install was fixed with a simple apt-get update and upgrade, before re-running the grub installer.

I don't know when it happened, but Skype has blown up on me sometime over the past two months.

$ skype
  > skype: error while loading shared libraries: libXss.so.1: cannot open shared object file: No such file or directory
  

Hmmm.

$ locate libXss.so.1
  > /usr/lib/x86_64-linux-gnu/libXss.so.1
  > /usr/lib/x86_64-linux-gnu/libXss.so.1.0.0
  

Okay, so it's moaning about the X screensaver library, libxss, not being there. But it is there, although specifically, it's a 64bit library. I bet Skype isn't 64bit...

$ file /usr/bin/skype
  > /usr/bin/skype: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, stripped
  

So that explains that then, we need compatible shared x86 libraries, but when you get these sorts of issues it's always best to see what else is missing. The ldd tool comes to the rescue.

$ ldd /usr/bin/skype
  > linux-gate.so.1 =>  (0xf76f2000)
  > libasound.so.2 => /usr/lib32/libasound.so.2 (0xf75df000)
  > libXv.so.1 => /usr/lib32/libXv.so.1 (0xf75d9000)
  > libXss.so.1 => not found
  > librt.so.1 => /lib32/librt.so.1 (0xf75d0000)
  > libQtDBus.so.4 => not found
  > libQtGui.so.4 => not found
  > libQtNetwork.so.4 => not found
  > libQtCore.so.4 => not found
  > libpthread.so.0 => /lib32/libpthread.so.0 (0xf75b4000)
  > libstdc++.so.6 => /usr/lib32/libstdc++.so.6 (0xf74c9000)
  > libm.so.6 => /lib32/libm.so.6 (0xf749e000)
  > libgcc_s.so.1 => /usr/lib32/libgcc_s.so.1 (0xf7480000)
  > libc.so.6 => /lib32/libc.so.6 (0xf7306000)
  > libdl.so.2 => /lib32/libdl.so.2 (0xf7301000)
  > libX11.so.6 => /usr/lib32/libX11.so.6 (0xf71cb000)
  > libXext.so.6 => /usr/lib32/libXext.so.6 (0xf71b8000)
  > /lib/ld-linux.so.2 (0xf76f3000)
  > libxcb.so.1 => /usr/lib32/libxcb.so.1 (0xf7198000)
  > libXau.so.6 => /usr/lib32/libXau.so.6 (0xf7194000)
  > libXdmcp.so.6 => /usr/lib32/libXdmcp.so.6 (0xf718d000)
  

So looking at this, I do not have both compatible libxss and misc qt libraries installed.

With Ubuntu, to enable multiarch support, check the file /etc/dpkg/dpkg.cfg.d/multiarch. There should be a line like below (and if there isn't, add it):

# in file /etc/dpkg/dpkg.cfg.d/multiarch
  foreign-architecture i386
  

Now we need to satisfy Skype's i386 dependencies.

 $ sudo apt-get install libxss1:i386 libqtcore4:i386 libqt4-dbus:i386 libqtgui4:i386
  

If we re-run ldd we can see Skype's shared library dependencies are now satisfied.

$ ldd /usr/bin/skype
  > linux-gate.so.1 =>  (0xf7749000)
  > libasound.so.2 => /usr/lib32/libasound.so.2 (0xf7634000)
  > libXv.so.1 => /usr/lib32/libXv.so.1 (0xf762e000)
  > libXss.so.1 => /usr/lib/i386-linux-gnu/libXss.so.1 (0xf7629000)
  > librt.so.1 => /lib32/librt.so.1 (0xf7620000)
  > libQtDBus.so.4 => /usr/lib/i386-linux-gnu/libQtDBus.so.4 (0xf75a6000)
  > libQtGui.so.4 => /usr/lib/i386-linux-gnu/libQtGui.so.4 (0xf6ae0000)
  > libQtNetwork.so.4 => /usr/lib/i386-linux-gnu/libQtNetwork.so.4 (0xf69a4000)
  > libQtCore.so.4 => /usr/lib/i386-linux-gnu/libQtCore.so.4 (0xf6702000)
  > libpthread.so.0 => /lib32/libpthread.so.0 (0xf66e7000)
  > libstdc++.so.6 => /usr/lib32/libstdc++.so.6 (0xf65fc000)
  > libm.so.6 => /lib32/libm.so.6 (0xf65d2000)
  > libgcc_s.so.1 => /usr/lib32/libgcc_s.so.1 (0xf65b4000)
  > libc.so.6 => /lib32/libc.so.6 (0xf6439000)
  > libdl.so.2 => /lib32/libdl.so.2 (0xf6434000)
  > libX11.so.6 => /usr/lib32/libX11.so.6 (0xf62fe000)
  > libXext.so.6 => /usr/lib32/libXext.so.6 (0xf62eb000)
  > libQtXml.so.4 => /usr/lib/i386-linux-gnu/libQtXml.so.4 (0xf62aa000)
  > libdbus-1.so.3 => /lib32/libdbus-1.so.3 (0xf6261000)
  > libfontconfig.so.1 => /usr/lib32/libfontconfig.so.1 (0xf622b000)
  > libaudio.so.2 => /usr/lib32/libaudio.so.2 (0xf6211000)
  > libglib-2.0.so.0 => /lib32/libglib-2.0.so.0 (0xf6118000)
  > libpng12.so.0 => /lib32/libpng12.so.0 (0xf60ee000)
  > libz.so.1 => /usr/lib32/libz.so.1 (0xf60d9000)
  > libfreetype.so.6 => /usr/lib32/libfreetype.so.6 (0xf6041000)
  > libgobject-2.0.so.0 => /usr/lib32/libgobject-2.0.so.0 (0xf5ff2000)
  > libSM.so.6 => /usr/lib32/libSM.so.6 (0xf5fe9000)
  > libICE.so.6 => /usr/lib32/libICE.so.6 (0xf5fcf000)
  > libXi.so.6 => /usr/lib32/libXi.so.6 (0xf5fbf000)
  > libXrender.so.1 => /usr/lib32/libXrender.so.1 (0xf5fb4000)
  > libgthread-2.0.so.0 => /usr/lib32/libgthread-2.0.so.0 (0xf5fad000)
  > /lib/ld-linux.so.2 (0xf774a000)
  > libxcb.so.1 => /usr/lib32/libxcb.so.1 (0xf5f8e000)
  > libexpat.so.1 => /lib32/libexpat.so.1 (0xf5f64000)
  > libXt.so.6 => /usr/lib32/libXt.so.6 (0xf5f08000)
  > libXau.so.6 => /usr/lib32/libXau.so.6 (0xf5f04000)
  > libpcre.so.3 => /lib32/libpcre.so.3 (0xf5ec4000)
  > libffi.so.6 => /usr/lib32/libffi.so.6 (0xf5ebd000)
  > libuuid.so.1 => /lib32/libuuid.so.1 (0xf5eb7000)
  > libXdmcp.so.6 => /usr/lib32/libXdmcp.so.6 (0xf5eb0000)
  

Skype now starts up and behaves as I would expect.

Like with Apache, with Varnish 3 you can test the syntactic correctness of your VCL files without having to roll some dice restarting a running server.

$ varnishd -C -f /path/to/mysetup.vcl
  > ...
  

Varnish will compile the file and print out it's full configuration as output. If there's an error, varnish handily gives you a compiler error to help trace what went wrong, which could look something like this

> Message from VCC-compiler:
  > Expected an action, 'if', '{' or '}'
  > ('input' Line 82 Pos 6)
  >     vcl_hash(req.http.Cookie);
  > -----########------------------
  >
  > Running VCC-compiler failed, exit 1
  

I've been using Oneiric since its release and in many ways it is a giant leap forward from Natty Narwhal. Unity is still horrible mind, but a lot of the rough edges (multiple / external monitor support) have been softened and the overall package is more stable.

Some things still grate, particularly the global menus and the overlay scrollbar, but they are manageable.

A great resource I think to checkout for anyone using Oneiric is webupd8. You can find a lot of tips to bend Oneiric to your will.

For me, that means getting an official JVM installed Installing Oracle Java 7 and getting a sane desktop environment configured with global menu and overlay scrollbars zapped away Things to Tweak After Installing Oneiric.

Linux Mint (now on release 12, Lisa) is increasingly becoming a viable alternative to Ubuntu. I've played around with it a fair bit, but I still think it is not quite there yet in terms of 'just working' when compared with Ubuntu.

For someone that needs to get work done with a minimum of fuss, Ubuntu is still the leader. I do hope though, someone can work out how to marry usability and power in a slightly better way than I feel Canonical are managing at the moment.

I first came across the Imagemagick library and toolkit when I was new to Linux and trying to satisfy Enlightenment and E-term dependencies and get them up and running. A task, and this was the mid 90s, I do not recall fondly. (And thinking about that experience, despite all the controversy surrounding Gnome 3, and Unity, the Linux Desktop has come a long way).

Anyway, unlike either Enlightenment or E-Term, Imagemagick has been a firm friend ever since.

There are three main tools I find I fall back on, time and time again.

Display, Convert, and Identify.

Display, funnily enough, will open up an x-window with the contents of an image.

Convert, perhaps the most useful tool of them all, lets you resize, transform, and re-format images.

And lastly, identify, lets you get information about a file, size, format, colour depth, offset and so forth.

The power of these tools is best wielded with simple loops in bash, or using GNU Find. For example to convert a bunch of images from 1920x1080 to say 1280x720, you could do something like this

$ for IMAGE in *.jpg; do
  >   # usage convert <action> orgfile.jpg newfile.jpg
  >   convert -resize '1280x720' $IMAGE $(echo $IMAGE | sed 's/.jpg$/-resized.jpg/')
  > done
  

Convert tries to be clever and maintain aspect ratios. In this case, 1920x1080 fits into 1280x720 maintaning a 16:9 aspect ratio. But what if we had a 1920x1200 input image? In these cases, convert would actually resize the image to something like 1280x800. But if we really want it to ignore common sense and squish things down to our requested 1280x720 you need to use the bang operator e.g.

$ convert -resize '1280x720!' srcimg.jpg dstimg.jpg
  

If you want to add a unix user to a supplementary group (say for example user 'aaron' belongs to group 'aaron' but I want to add him to the 'wheel' group as well) you use the usermod command.

$ usermod -a -G wheel aaron
  

The -a argument is very important, it ensures arguments to -G append to the existing list of groups. Otherwise existing groups will be replaced with the argument supplied.