Peter Joseph Flanagan, Family