Hey, first time posting here... You are my height, but I'm about 148lbs and wear a 38 in jackets. I actually have the ashby in xsmall and have worn it over sweaters. It's a bit snug for me around the chest, but fits perfect with just a shirt under. I should have gotten a small but Xsmall worked out for me and I like the slim look. It sounds like an Xsmall would suit you. For reference, the chest (pit-to-pit) on a small ashby is 20.5 inches. Hope this helps.
I have an oppsited opinion. I'm about the same height and weight as yours, with a 36 chest and I've found small Ashby fits me quite well, not baggy nor slim. If I size down to xsmall the jacket won't be as practical as it's designed to be.
If you're looking for a more fitted look, I'd recommend you try their international line.