Joseph Martin Scalise, Family