In fact the VAST majority of the world doesn't follow Islam. So they aren't told to lower their gaze.