It depends how self-conscious you feel. Unfortunately many men here will stare no matter what you wear.
My wife dresses fashionably. Just because some men will look doesn't mean she shouldn't buy and wear clothes she likes.

She likes jeans, skirts, t-shirts, blouses, boots, sandals and she wears them all, but not together.

Also, dress according to your surroundings.

If the men stare they're the ones who have the problem and lack the social and cultural maturity, not you.