James Brian Templeton, Family