Thomas Dodge Hathaway, Family