Magento is a complicated piece of ecommerce software that has been in the past notoriously difficult to employ TDD practices on. Luckily in the past few months we've had Alistair Stead's Mage_Test and now Ivan Chepurnyi's EcomDev PHPUnit Suite testing frameworks released, to make this a little less difficult.
On my most recent project I exclusively employed Mage_Test for my Unit Testing and with a few exceptions, it's performed extremely well.
Mage_Test takes a very hands-off, light-weight approach. EcomDev meanwhile, on the surface, appears to provide far greater support for testing at the expense of some complexity.
Installing EcomDev's Test module is easy. You can install it from Magento Connect or you can get the module directly from the developer's Subversion repository
$ svn co http://svn.ecomdev.org/svn/ecomdev-phpunit/trunk
Once you have the code, it's just a matter of copying into your magento store as with any other module:
$ cp -r <ecomdevtestdir>/* path/to/your/magento/rootdir
Once the files are in place, you'll need to edit app/etc/local.xml.phpunit and supply some details for a test database connection and some path information relating to the store site root URIs.
Once your database and paths have been defined initialise the test database by changing into your store root and running the UnitTests.php test suite
~/Sites/my/store/root: $ phpunit UnitTests.php
This will take some time (one of my clients has a 400meg database and a lot of orders, it took ~ 4 minutes).
You're now ready to start writing some tests.
For more information on EcomDev's PHPUnit module see this blog post or get a copy of the (very) comprehensive EcomDev PHPUnit 0.20 Manual