I believe his question is... Family should leave before him or not?

Yes, you can leave together,,, once your visa canceled you should leave within a week.

best option... just get in to an immigration office and reconfirm it.