James Lawrence Fallon, Family