Absolutely, the one thing that you should spend the most on is speakers. The differences made by any other part of your system pale in comparison. The next biggest contributor is room acoustics.
Believe me, I'm of a very analytical and subjective nature, so I will not be imaginig hearing things.
Just thought I would bring the digital side of my music up to scratch.