If I had a 30 year old neighbor who built a fun fair in his yard, invited young kids to play with him every day, and regularly slept with them in the same bed overnight, I wouldn't want anything to do with him.

Whether or not he had intercourse with the kids would be immaterial - there is something a bit 'not quite right' about him, and about the parents who were happy enough to let their kids share his bed.

Jackson wass ostracized not (just) because of his clothes, nose or pale skin. He defied social norms through his interactions with young children.