John Jay Negley, Family