FWIW, that's just my personal preference. I've seen Lyo (from PJT Sydney) wearing his seersucker a couple times and I think he executes it fairly well.
The key would be to strip away seersucker's traditional stylistic connotations. I recall a photo from several Pittis ago showing Kamoshita wearing a seersucker rig in a decidedly Neapolitan way.
If you're into it go for it, I'll stick to drill cotton