greentea - Yes, a lot of people "jailbreak" factory unlocked (or even locked to carriers) iphones to install apps from Cydia. Just be cautious about what you install from Cydia though. Some themes and other stuff available on Cydia have caused stability issues. In general, there is nothing wrong with Cydia and most of the apps available on it.

And you can get rid of it any time by doing a full restore.