I'd change the buttons to gold or brown horn (I prefer the latter). In fact thats what I did with mine. If you don't want to, I wouldn't wear it as an odd jacket since it will look like an orphaned suit jacket.
Another difference between suit and sport coat is that the latter can have 2 or 3 patch pockets, making it casual. You don't want to have patch pockets on suits.