I can appreciate the concern about SW&D, but it's a pretty big tent over there, from my own lurking, and I wasn't suggesting posting there yourself. I just meant that you need to find what you like.
Perhaps FrankCowperwood is a good point of reference for you. He wears a lot of Epaulet pants in a casual way. Cotton Dockers has a very different vibe, with a lot of Engineered Garments stuff. Our own IS has some great casual looks, his latest reminding me very much of my youth (Madchester, Happy Mondays, Stone Roses, Inspiral Carpets, etc.). Other names worth looking up are Chullilo, Crat and NOBD. Chullilo wears a lot of chinos and casual shirts and Crat and NOBD need no introduction.
This thread is moving pretty fast lately. I quoted SeaJen because I think he has identified some great examples of people who do casual very well and likely consistent with what Noodles is aiming for. I also think the NMWA thread is pretty good for ideas on casual clothes.