Although guilty on numerous occasions of donning this look, I also feel it is "not quite right." I think that wearing a properly fitting shirt and pants help make the look a little less "cell phone salesman in the mall" (not that there's anything wrong with selling cell phones), but nevertheless, it is still incomplete in my humble opinion.
Speaking from my personal aesthetic sense, I feel that wearing a v-neck/crewneck sweater or a cardigan helps “complete” the look; although not as fully as a blazer or suit jacket. Living in an area where the weather is sweater worthy 8 months of the year, and also being too thin to find a properly fitting blazer, I usually try to utilize that look as much as possible. When doing the whole shirt, tie and no jacket thing, I try to wear something like a blue gingham pattern shirt and a knit tie. This way, I won’t be mistaken for the IRS.