Robert James Pitman, Family