Charles Norman Hughes, Family