Have you actually tried talking to him about how he feels he's doing as a father? Does he even want to be a Dad? Some openness and honesty, without attack or fear of repercussions would be sensible.

It sounds to me like he's never really bonded with your child. You say you fear leaving them alone for time, but I suspect that's exactly what is needed, to actually build a relationship.