These Pagans, who existed at the time of our Prophet, were different from other non Muslims. They used to be a barbaric group of people who did much crimes including burying alive female children who they deemed inferior to men. It was due to their crimes and evil that they were deemed "unclean" and not their actual physical cleanliness.