You came here asking for advice and advice was given. You may choose to go with it or disregard it, but you must remember that style is something that's quite subjective in nature. That said, you can wear what you like -- after all, comfort must come first. However, if you are looking to wear a scarf with your jacket, I'd recommend getting a top-coat or a peacoat or an overcoat that would look good with the scarf.
In fact, Will talks about how men should wear scarves in his blog - "Wearing a Scarf