Versus – A web-based tool to create experiments based on a 2AFC methodology

August 24th, 2016


Versus is a web-based application that allows experimenters to create multiple experiments based on a full 2AFC methodology (two alternative forced-choice). Participants are presented two images side-by-side, and — without imposing a time constraint — they are now asked to choose one of the two images.

mturk

The preliminary result of the experiment should be a ranking between all images presented to the participant. To achieve a full ranking, each image should be compared to each other, i.e. for n images, n(n-1) comparisons are required per participant. Silverstein and Farrell, (2001) proposed a method to reduce the number of comparisons to nlog2(n) using a binary tree sorting method, see Fig. 1 in the paper. The idea behind this: If image A is rated higher than image C, and image B is rated lower than image C, then A does not need to be compared to B, i.e. the ranking is then A > C > B.

The web-based tool aims to provide an easy-to-use interface to create and define 2AFC experiments, facilitating the integration to Amazon’s Mechanical Turk to recruit participants in future versions of the tool. Furthermore, we plan to facilitate data analysis using R, as well as providing a tool to visualise the experimental results.

 

This project is a collaboration with Queensland University of Technology.

References:

  1. Silverstein, D. A., & Farrell, J. E. (2001). Efficient method for paired comparison. Journal of Electronic Imaging, 10(2), 394-398.
  2. Heinrich, J., Kaur, S., & O’Donoghue, S. (2015, September). Evaluating the Effectiveness of Color to Convey Alignment Quality in Macromolecular Structures. In Big Data Visual Analytics (BDVA), 2015 (pp. 1-8). IEEE.
  3. Heinrich, J., Vuong, J., Hammang, C., Wu, A., Rittenbruch, M., Hogan, J., Brereton, M. & O’Donoghue, S. (2016, May). Evaluating Viewpoint Entropy for Ribbon Representation of Protein Structure. In Proceedings of the 37th Annual Conference of the European Association for Computer Graphics. The Eurographics Association and John Wiley & Sons Ltd.