James Haining Lou, Family