I wear the vests with button down shirts and never with jeans. Always with slacks, fatigues, chinos, etc. not necessarily with a tie, but ties help. Throw a jacket or even a cardigan over if weather permits.
But remember, no rules. Work it out with your wardrobe. Could work with a tee. I've worn blue vests with a blue tee underneath.