I'm going to second that; I've worn it with navy, grey, white, taupe, you name it. Here I am rocking it at a garden party last summer. One of my favourite squares ever (and this is from a guy who designs pocket squares).
I can't speak to button-downs because meh. For the popover, I'd do 1.75" back height, 3.25" collar points in a semi-spread, a stiff neckband but soft/unfused collar. That way it will be soft and look casual but will still hold its shape if you want to wear it under a jacket or something.