Gerard Brennan Harvey, Family