I go sockless all summer except when I am wearing a suit for work. Rotate your shoes, clean your feet, you will be fine. If it's a problem because you sweat a lot or whatever, then wear socks. Some guys always wear undershirts for the same reason. I don't wear those either.
As for loafer socks, either wear socks or don't. You are not fooling anyone.