Julian Randolph Fox, Family