Benjamin Swartzman, Family