Charles Joseph Morris, Family