James Ernest Simon, Family