Patrick Jean Rogers, Family