Thomas Edward Walley, Family