Which Panerai are you looking at?
The Portuguese has a 41mm dial, but I'd argue that it wears pretty big since there's practically no bezel.
If the Portuguese works for you, I think a PAM should be no problem.
Of course you'll never know till you've tried it on.
I was comfortable with my 36mm Datejust, but I had no problems transitioning to 40mm (Sub) or 42mm (Speedy), so the jump from 41mm to 44mm might not be that big a deal for you. Especially considering the IWC wears...