BOTTOMLINE: if you're not interested, just tell him you're not interested...that's it.

You have to stand firm and tell him you prefer to keep your working relationship on the professional level.

Stop whining...no one could help you but yourself.