Punching and/or behaving violent might have very well backfired and you may have ended up on the wrong side of the law.
By complaining to the authorities concerned, hopefully you will get your justice.
Certainly that guy was a jerk, and kudos to you and indeed your wife, a lady after my heart, for maintaining dignity and decorum even in the face of such dastardly behavior.