Assuming you are a female, then you should take the matter seriously. You made a mistake by entertaining his queries. You should have simply not responded to him at all or walked away. You should have threatened him by telling him you would call the police if he still insisted on talking to you. This is what most females do in such situations.
Here is what I suggest you should do now. First, stop going to that restaurant if possible. Second, if you still wish to visit the same place, avoid talking to him. You can also bring the matter to the attention of the restaurant Manager. They will help by getting involved. Third, if he stalks you to another restaurant, take his picture on your mobile and tell him on his face you will report the matter to the police. If still does not come to his senses, then you can report the matter to the police. They will help you out.
If you don't take a tough stand now, he will continue to get more cozy with you. Remember, it takes all sorts of people to make up this world. You need to be strong to live.