There is something seriously wrong in your man. He appears to be one of those men who is just not content with a single woman in his life even after marriage. After reading what you have stated, I think it is time for you to take some hard decisions. Just kick him out of your life. It is better to suffer for a day or two rather than suffer each day of your remaining life due to a person who does not understand the meaning of loyalty to a wife.