Eli Veil Hammond, Family