Eugene Philip Yale, Family