It's inappropriate behaviour. I also think it's borderline sexual harrassment. I personally have no qualms getting inside a lift with a man.

But in this situation, I would rather wait for the next lift. One way to stop the harassment? Wear a wedding band. and call a fake tel. no and pretend it's your husband. Failing that, tell them straight that they will be reported to the police.