James Michael Gardener, Family