As long as I've lived I ALWAYS understood that it was ME who was the infidel in the (your) eyes of Islam ... as in *I* am the unbeliever.

I've never heard it said the other way around. :D