Audrey Elisabeth Oliver, Family