You are both working parents and that puts a different complexion on the matter. I used to work in a playgroup where mothers would arrive with their three-year-old and say "I can't do anything with him, you sort him out." I can see now that you are not that sort of parent.
The person who has most influence on him will be the one with whom he spends most of his waking hours so best thing you can do is to make sure that this is the right person and then you won't need anyone extra. I read somewhere that the best thing for the development of a young child is the company of an adult who is interested in him. Is his carer interested in him? Does she talk to him (and in what language?), read to him, play games with him? You are clearly trying to do your best. Would it be possible for your wife to work shorter hours, at least until your son is old enough for nursery or school?