depends on how baggy and how it looks
i have a coat that fit great without a jacket underneath but decided to size up since i wanted to pull double duty and be able to wear it over a jacket as well
i do wish i had just gotten the smaller size but i still wear the coat by itself when i can
you can see that the shoulders look a little big here
but it's not terribly big
also i either keep the coat open the way it is in the picture... or buttoned but with my hands in the pocket kinda holding the coat tighter around my body to avoid showing how baggy/roomy the body of the coat is without a jacket underneath Warning: Spoiler! (Click to show)
The problem here is that your coat ends mid thigh. He's talking mid calf. That's a lot harder to do without looking baggy and bad...