what you are talking about is Jama3a "grouping in prayer,
but for Jum3a it is according to what system you are using "Mathhab"
for Shafii and Hanbali , it should be 40 men
for Malki it should be 12
for Hanafi it can be done by 3 only but one of the three should be able to make imamah.
http://www.islamweb.net/ver2/Fatwa/ShowFatwa.php?lang=A&Id=7637&Option=F...
if you can read arabic