The main thing is that it should not look like you just forgot your pants. So, you have 3 ways to go: 1) More casual fabric (wool jersey, tweed, etc...) 2) More casual or aggressive cut and details (raw hems, stand up collars, etc...) 3) Both of the above. I usually prefer both of the above, but that's a personal preference. Unless you want to look like a guy from a bad GQ shoot. Which is fine, except that you'll generally look terrible - neither classically excellent nor particularly modern and cool. Just an exercise in mediocrity.