it's up to you how to take it.
there is no "right" and "wrong" between two.
If you throw him out him after this affair, it's totally perfect. If you accept it, it is also just fine..

*********************
"What lies behind us and what lies before us are small compared to what lies within us."
Oliver Wendell Holmes