you should reply them with Bahasa Indonesia..or Javanese...or Sundanese...or Maduranese..or the rest of 360 tribal language in Indonesia.
Take it easy, take the positive: they may need to exhaust their tiredness speaking in English the wholeday.
I experienced often and I reply them with Bahasa Indonesia then I said I'm an Indonesian...smile.
But I do agree with you to come ask first where do we come from..