Nancy Lea Haley, Family