I wear v-necks with button down collar shirts all the time. Cardigans less often but they make it into the rotation. Generally without a sport coat, but with a windbreaker, paired with slim chinos or 5 pocket cords and chukka boots. It's that late 60's west coast ivy look - Paul Newman, John Cassavettes, Steve McQueen. I feel like I've been dressing that way since high school.