It is better to apply for PAN card even if you are an NRI

PAN card is required for registering property and many other govt transactions.

Just because you have a PAN Card doesnot mean you have to file tax. there is seperate form to waive that.

You have worry only if you have taxable income from India, in which case you have to file tax anyways.