Actually, there is evidence of some Hitler sympathy on the duke's part. It's not clear how far it went, but it was there. He was not sent to the Bahamas because of the kidnap plot nonsense, which in any case no one in Britain knew about at the time, and which the British government did not take seriously when they did find out. He was sent because Churchill couldn't stand him. Churchill also thought that because his past admiring comments about Hitler had not been retracted, they served as a rallying point for certain elements of British society who wanted to make a deal with the Nazis and used the duke to paint their own appeasement as still somehow respectable.