When using apt-get remove or aptitude remove the configuration files of the packages uninstalled are not deleted.
Now you can manually do it yourself with apt-get purge
dpkg --get-selections | grep deinstall | sed 's/deinstall/\lpurge/' | dpkg --set-selections; dpkg -Pa