Stop working and start discussing some other non work related discussions until he disappear and allow you to work alone.