Seth Forrest Gorman, Family