Sounds like you really don't have any type of relationship with the friend any longer.
Granted, if he finds out it might be weird if you run into him unexpectedly, but who really cares? Maybe you'll actually get some great scene out of a movie where you see him in a nice restaurant, or a boardroom, or a baptism, and he screams "I can't believe you slept with my MOM!!!!", leaving everyone in the place to stare at you with wonder.
With that said, assuming you're single, why on earth wouldn't you pursue that? What a great story you'd have to look back on in the future.