Shape Model Evaluation

ShapeWorks support quantitative evaluation of shape models, which can be used for algorithmic parameter tuning. Quantitative metrics are generalizability, specificity, and compactness. In Examples/Python/, run python -use_case ellipsoid_evaluate to see an example of how to compute these metrics.

Evaluation metrics


For fixed training data, a compact model pdf should describe the data's distribution using the smallest possible number of parameters.

Poor Model (compactness = 0.3)

Compact Model (compactness = 0.99)


The model should be able to generalize from the examples given in the training set, hence describing any valid instance of the class of object, not just those seen in the training set.

Generalization metric


This is the requirement that the model can only represent valid instances of the class(es) of objects presented in the training set. Hence, the model is specific for this training set.

Specificity metric

What is a good shape model?

A good shape model should balance the trade-off between three requirements, namely specificity, generalization, and compactness. In particular, a shape model needs to generate samples that are plausible (i.e., respect the population statistics). It also needs to generate samples beyond the training data (i.e., generalizable) while describing the population with few parameters (i.e., compact).

ShapeWorks Commands

shapeworks readparticlesystem --name *.particles -- compactness    --nmodes 1
shapeworks readparticlesystem --name *.particles -- generalization --nmodes 1
shapeworks readparticlesystem --name *.particles -- specificity    --nmodes 1

Model Evaluation in Studio

Under construction, coming soon!