John Lloyd Adams, Family