I kind of agree with you. I said I can 'object' to criticism when based on a false premise. I don't tell people to shut up and not say such things, I just usually take the p*ss. Or alternatively I just ignore some comments and move on.

I don't go to mythical places with strange men.
-- Douglas Adams, The Long Dark Tea-Time of the Soul.