Anthony Thurgood Walker, Family