Michael Scott Jarvis, Family