I was never a big fan of olive - and there's many a thing called olive - I'd mainly seen it as an American non committal style.
But I have a cotton suit which I like and gradually came to see as possibly olive rather than light brown.
Its probably best described as olive with a brown caste rather than a green caste. Its the green I don't particularly like.
Like black - its black with green caste that looks the worst.
Some old pics.