Next time just get up and say
'gotta go...my husband's waiting at home with the kids... and he gets nervy 'cos he is only just out of prison'.