Charles Rosenberg, Family