Samuel Arthur Sue, Family