Emma K Libby, Family