Thomas Vincent Grimes, Family