The whole point of paying a deposit to reserve something is so that you are committed to the agreement. If you change your mind then their would be a financial penalty. I'm sure understood that before you agreed to pay him.