Mainly because herbal teas like Mint/peppermint are consumed to aid digestion. Therefore the addition of milk, specially if you are even mildly lactose intolerant or suffer from gastritis/gastric related problems,or even diarrhea, will take away that benefit and can very well worsen your symptoms. Also adding milk=adding calories.
To reap their full benefits, herbal teas are mostly best drunk on their own.