Robert Walter Stanley, Family