Charles Joseph Leoni, Family