Yasuhide Elmer Watanabe, Family