Personally I would have gone back to Wordpress as theres no reason it should be slower, its much faster I think. Joomla is a (too) heavy and complicated CMS.
I would have said this too, but just offering the help. Small site - Wordpress and there is probably an addon or issue or setup issue causing it to be slow. Its FAST. Joomla vs Wordpress for a small site or just content driven - Wordpress. Joomla comes into its own with a larger website and around where drupal is before you head into the silver stripes, expression engine, syphony and then beyond that.
http://symphony-cms.com/ on a side note I really fancy giving a go. It is an xslt driven CMS and the company my brother works for uses it as their main CMS. My brother actually designed the site you see there.
Sooph, if you read the link you can check if the module is assigned and the tag in the template etc.