Thomas Hadley Hamilton, Family