Patrick Gerald Cherry, Family