Adam D. Schwartz, Family