Depends on where you go on the Vineyard. Edgartown is very no-socks, Republican. Oak Bluffs is old African-American aristocracy, lots of cool Victorian houses. Vineyard Haven is a bit earthier, although pleasant enough. Then you get Chilmark, Menemsha, etc., further south which are much more away from things. Off-season, of course, you might find the whole island deserted except for construction crews.
Nantucket used to be more remote than the Vineyard, but that could have changed b/c of the bull market and increased prevalence of private jets.