and listen...but the other person knows I don't mean it as calling him "my sweetheart".

My sentence usually starts with "Listen good, habibi..." coz sometimes, they don't clearly understand what I say when I speak ENGLISH too fast for their understanding.

I agree, people should be cautious with what they say as this may be understood differently. Try "sadeeq" instead...

:-)