Joseph Edward Sullivan, Family