I think some of the parts for the '05's are different from '06 & later - power steering reservoir comes to mind as well as the radiator, & I know from looking at the top components in the "bucket" in the trunk that they seem to be different than later years (at least wiring-wise). I don't know if you can upgrade all the parts that are different, but since some parts are hard to find for '06 & later, that's just going to be worse for '04-'05.
Also, '04-'05 (& some '06's) are subject to the "melting DRL" problem, but if you start keeping your headlights on before any damage is done that prevents problems.
I personally would stick with '06 & later, BUT there are plenty of people on here with '05's who are doing fine with them.