Yeah, ties shouldn't go with such a casual bottom half and no sport coat. The only kind of shirt (OCBD) that really goes with that bottom half shouldn't really have a tie with it to begin with. If you wear a dressier shirt, like a spread collar it will look really bad with that bottom, especially if you have a tie on with it.
My personal opinion is that if you are wearing a tie you must have a jacket of some sort on. The whole tie with no jacket is a bad look.