You just answerd your own question: Who sets the limits on free speech?
Stealth - give me an example of speech that I could not say in the UK that does not threaten to physically harm someone. Then we can have a discussion on what is and what is not free speech.