Edward Gerard Summers, Family