I think for visa application and renewals the passport would for sure be given back or if they dont want to hand it over to the concerned staff at any time then they may as well send someone along for all those actions. Anyways, it can be seen both ways...illegal or care for its safe custody, though I would still say let it be with the individual even if it is not safe with him.