Brook Miller Foster, Family