William Peter Harvey, Family