Happened to me 4-5 times, I don't allow it. I'd just get out of my car with a "special" face, I'm kind-of scary.

He walks away and I take the space.