Normally as per the business etiquettes the advance given is yours if the party is backing out from the deal. It is upto you to decide to give it back to him as a goodwill gesture as he might be financially poor.