Benjamin Joseph Ramsey, Family