It is always appropriate to wear a hat! The type of hat is more of the question- if you are wearing one out in the evening you probably want a shorter brim so it is easier to see people in a bar or restaurant (i.e. not a floppy sun hat).
A cheaper hat is great if you are putting it on over wet hair at the pool or running around on your bike and just need it to last for a season. Spend more money or go for a good quality vintage hat if it is your "signature" piece and you are wearing it on a frequent basis. A hat sits on the top of your head and is the first thing that people notice so you want to make sure it expresses you well.
At The House of MacGregor custom hat shop, people try on a variety of different shapes and styles to find the one that works on them. They measure their customer's headsizes and pick out the materials together to get a custom fit they feel comfortable in. Most of my customers that say they don't have a "hat face" but they simply have a hard to fit headsize and have not had a hat on that fit them before. You want to feel comfortable in your hat! To have a hat made just for you, check out http://www.thehouseofmacgregor.com