McCall Miller Branch, Family