Well the Fab Four used to wear them with light grey or black suits, as well as with jeans.
It's a self-confidence thing mate.
to be brutally honest, if you feel like a twat wearing them, then you'll probably look like a twat wearing them. Sorry, but that's how it is with clothing really. If you don't feel comfortable in something then you won't look ditto!