There are those who maintain that a buttondown collar makes a shirt inherently casual. Some of these people point to the buttondown collar shirt's origins as a sport shirt (for playing polo), in support of their point of view.
Other people maintain that a buttondown collar shirt is not limited to casual wear, and can work perfectly well with a suit and tie. Many of them note that this is a look long popular in certain trad/prep circles.
So, one is free to side with either take on this one. Just pick a side, and you're golden.
Ultimately, one's attitude toward such things may depend on factors which include geographic location, family social and economic background, profession, age, etc. In this it's rather similar to the question of whether one can wear loafers with a suit and tie.
Me? I consider a buttondown collar shirt to be fine as a casual shirt (say, with a pair of chinos, and without a tie). I also find it to often be acceptable with a blazer or sport coat. And sometimes - but only sometimes - acceptable with a suit. For example, I tend to regard it as more casual than most other types of dress shirts, so I tend to avoid it if I want an extremely formal look. But if I'm wearing a seersucker or tan poplin suit to a relatively casual social outing, I might find such a shirt to be perfectly appropriate. (Just like my take on loafers.)
And, of course, not all buttondown collar shirts are created equal. Fabric, color/design, style, etc., can swing a shirt more to the casual end of the spectrum, or more to the formal end. (Again, just as all loafers are not created equal.)