Hahahahah...go to low level people areas like the Souq (Sofitel Area); of course you get mostly labourers to stare at you and your family. Its the same anywhere in the world as far as i remember!

On top of that, if your wife does not look like from the same country as you. Then it becomes bigger treat for them and a premium reason to stare at you.