Swarm jacket didn't work out for me either. Great jacket, great fit but didn't fit with my overall style.
I did pick up a boucle sweater and a pair of the cast 2 obsidians. I have a 29 in both the cast 2 coast and zincs and had to size down to a 28 because of what was available. Definitely tight at first but they have stretched out a good deal and now fit great.
wish they made the all blacked out in the summer rider as well
did you take same size in JE tops in medium as your rider?