Shape Model Evaluation
ShapeWorks support quantitative evaluation of shape models, which can be used for algorithmic parameter tuning. Quantitative metrics are generalizability, specificity, and compactness. In
python RunUseCase.py -use_case ellipsoid_evaluate to see an example of how to compute these metrics.
For fixed training data, a compact model pdf should describe the data's distribution using the smallest possible number of parameters.
Poor Model (compactness = 0.3)
Compact Model (compactness = 0.99)
The model should be able to generalize from the examples given in the training set, hence describing any valid instance of the class of object, not just those seen in the training set.
This is the requirement that the model can only represent valid instances of the class(es) of objects presented in the training set. Hence, the model is specific for this training set.
What is a good shape model?
A good shape model should balance the trade-off between three requirements, namely specificity, generalization, and compactness. In particular, a shape model needs to generate samples that are plausible (i.e., respect the population statistics). It also needs to generate samples beyond the training data (i.e., generalizable) while describing the population with few parameters (i.e., compact).
shapeworks readparticlesystem --name *.particles -- compactness --nmodes 1 shapeworks readparticlesystem --name *.particles -- generalization --nmodes 1 shapeworks readparticlesystem --name *.particles -- specificity --nmodes 1
ShapeWorks Python tools
# Read the particle files from a pre-trained shape model particle_data = sw.ParticleSystem(<list of local point files>) #evaluation tools shapeworks.ShapeEvaluation.ComputeCompactness(particleSystem=particle_data, nModes=1, saveTo="scree.txt") shapeworks.ShapeEvaluation.ComputeGeneralization(particleSystem=particle_data, nModes=1, saveTo=save_dir) shapeworks.ShapeEvaluation.ComputeSpecificity(particleSystem=particle_data, nModes=1, saveTo=save_dir)
ellipsoid_evaluateuse case for detailed examples.
Model Evaluation in Studio
Model evaluation is performed in Studio automatically under the metrics panel. See Studio - Metrics Panel for more information.