How can I prevent this and help him to be a little more docile as my females are, without getting bit myself?

I have 4 mice, 3 are white females, and one greyish tan male. My male (Xavier) I had just gotten the third female and the male yesterday at Petco. He is a replacement for a black female that leaped out of her cage, and I have since returned because of my other animals. While I was holding him yesterday, he raced up to my shoulder where I calmly cupped him and tried to bring him down to level for fear of similar incident as female. When my husband tried to take him from me, he (Xavier the mouse) bit my husband. He shies away from contact and leaps over my hand when I try to pet him to get him used to contact. I'm very afraid of loosing him.