It probably doesn't make much difference but I have two brushes, one for darker shoes and one for lighter shoes. I also have one cloth per color: merlot, black, walnut etc...
But it takes a long,long time before a shoe can even change color from the polishes. I tried adding walnut polish to a dark brown shoe once for fun to see whether it will lightenup a bit, but it had no effect. I think it will take sometime before a shoe can actually change shades.
Basically, I wouldn't worry too much about cross color mixing in the brush or the cloth.
Unless I'm mistaken (now there's a hedge I should use more often), a good rule of thumb is to use different brushes/cloths for different colors -- especially when going from light to dark. In other words, using the same brush on your dark brown shoes as you use on your black shoes probably won't affect the black shoes, but using the same brush for black shoes and your walnut shoes will discolor the walnut shoes.
(I've heard this and it makes intuitive sense to me so I splurged and got a few extra $8 brushes from AE, but it might not be necessary after all.)