I think there's a difference between not abandoning her and a loving home. He could have easily have been raised in a loveless home. It would still affect him since he'd come into his power a different way and he'd have more of an understanding of love and family even if it wasnt' something he saw a lot of.

Emma//Hufflepuff//3