Knot + material + construction.
Knot: As Beatlegeuse commented, it MUST have a good knot when tied (note: this can also depend on what kind of knot you tend to tie/like). Just because it is expensive does not mean it will necessarily knot well.
Material: Stay away from blends. Look for all cashmere, all silk, all linen, all cotton, etc. Obviously, NEVER buy synthetic ties. Also, if a tie is made from 100% of the aforementioned materials but still feels flimsy or cheap to the touch, take a pass.
Construction: Depends on what you are looking for but generally ties come in 3, 5 and 7 fold construction. More folds does not necessarily make for a better tie, but it does mean more material was used in the construction, which generally makes it more expensive.
www.dieworkwear.com did a great breakdown on ties you might also want to look into.