It is your choice. I have lived here for over 3 years and I choose not to wear shorts and vest tops. I wear tracksuit trousers and t-shirt for walking along the Corniche.
It depends how you wish people to perceive you and whether you care about that.