Richard John Early, Family