I'd give fencing a go. The advantage of fencing is that you can do it anywhere (according to Hollywood) and no funny courts or funny balls are needed. For inspiration see "Princess Bride" which besides being the best film ever made (IMHO), also has the best sword fight / fencing duel (marginally pipping "Highlander").
In certain circles (rings), boxing fits into the "aristocratic" category. "I used to box for the army..." etc. Other games you might like to look into are: Hunting (horse and hounds) Real Tennis Raquets Eton Fives Harrow Football The Wall Game Polo but theses all need funny courts, horses and/or funny balls. Finally, you could look at Archery... if you've ever dreamed of winning an Olympic Gold medal but can't run, jump or throw things about, this might be the game for you!!