Robert Blake Hanley, Family