Any ordinary person will understand your unwillingness if you show your body language and ofcourse you can very well greeet him by words in order to avoid embarassing situation.