Yes, you are right. Just ignore him and his threats. If you legally right in your steps, there is nothing to worry about.
Good luck.