You just dont think only, you are perfectly correct. It has something to do with phonetics. When two vowels come together, they can only make a dipthong which usually belongs to a single word. It is phonetically impracticable. Just try to say the following sentence.
A apple a hour is always a amazing treat. You will see how difficult it becomes to say it in fast speech.

Whenever a word beginning with h is pronounced with a vowel (usually an a), 'an' and not 'a' has to be used. Whenever the consonant 'h' is used, only 'a' is acceptable and never 'an'.

... I take a high road, and you take a low road. [Bernard Shaw]

My kingdom needs an heir; I need a son. [Beowulf]