Yup. There are exceptions. Species-specific armor can only be worn by those species, and some species can't wear certain types. The Katarn armor from the Avenging Mort quest, though, can be worn by all species, including Ithorians and Wookies (go figure).
Mandalorian can only be worn by Commandos and Bounty Hunters as I wrote above.
And, finally, factional limitations mean that Imps can't wear Reb and visa versa.
But basically, yeah, anyone can wear any kind. (Jedi can't wear any armor, but they can wear Jedi-specific clothes that have a slight armor component.)