I would. I wear orange and red shirts too.
I guess it might be a skin tone thing. I'm brown so I look pretty awful wearing a white shirt in the winter since it makes me look sick. Blue oxfords are fine. I personally like houndstooth shirts as well as striped blue/white oxfords or check patterns as well.
As for me, I realized that you have to dress according to your skin tone, so what might work for someone who is white might not look as good on someone who is brown or black. I realized this when I started dressing for work when things would just look a little off if I was copying what I saw white 'models' wearing in menswear catalogs etc. Sure you can get by doing the super traditional thing. It works and looks great. However, if you're trying to develop your own sense of style, then why not try something else?