Many times we've been threatened with deportation over the years because we're "interfering".

Normally it means the person is doing something illegal/morally wrong and they know they've been caught.