AutoScoring – baby steps

Automatic computer analysis (or autoscoring) of crystallisation images is somewhat of a Holy Grail in crystallisation. After trying a number of different approaches to autoscoring, we are now collaborating with Patrick Hop of DeepCrystal to use a deep learning approach to image classification. The DeepCrystal analysis leads to a set of 13 probability vectors (which sum to 100%). The 13 classifications that are used by DeepCrystal don’t map onto the 16 scores that users can assign to an image, so although we save all 13 vectors, we use empirically derived cutoffs to find images likely to contain either crystals, or that are clear. These are then presented to users via See3 as autoscore “crystal” or “clear”. Autoscored crystal images have a pink border around them, and Autoscored clear images have a grey border around them.

The empirical cutoffs were determined by using an ROC curve comparing DeepCrystal scores with C3 scores for two sets of 12,000 images. To get a true positive rate of 80% (80% of images scored by a human as ‘crystals’ are autoscored as ‘Crystal’) we estimate we currently have a false positive rate of 40% (40% of the autoscored ‘Crystal’ images will not contain crystals). We have set the bar a little differently for autoscoring ‘Clear’ – a lower true positive rate but a correspondingly lower false positive rate.

Both user- and auto-scored images are used in the ‘Sort by Score’ option in See3. User scores are always presented before autoscores.

If the option of ‘Last User Score’ is selected on the Score pulldown menu, then only user scores are used to order images. Depending on when user scores were created, this can give unexpected results. The same plate as shown above is shown below, but this time with the ‘Last User Score’ as well as the ‘Sort By Score’ option selected.

This work is being done with Chris Watkins of CSIRO’s scientific computing team.