Paper: Benchmarking simulated robotic manipulation through a real world dataset
We present a benchmark to facilitate simulated manipulation; an attempt to overcome the obstacles of physical benchmarks through the distribution of a real world, ground truth dataset.
Users are given various simulated manipulation tasks with assigned protocols having the objective of replicating the real world results of a recorded dataset.
The benchmark comprises of a range of metrics used to characterise the successes of submitted environments whilst providing insight into their deficiencies.
We apply our benchmark to two simulation environments, i.e., PyBullet and V-Rep, and publish the results.
All materials required to benchmark an environment, including protocols and the dataset, can be found at the benchmarks’ website here.
3D plot of the bivariate normal distribution probability density function (x and y Cartesian positions) of manipulable object final positions across 20 repeats. An image of a single repeat of the final static resting pose for tasks 3 through to 10 is included.
Collins, Jack; McVicar, Jessie; Wedlock, David; Brown, Ross; Howard, David; Leitner, Jurgen. Benchmarking simulated robotic manipulation through a real world dataset. IEEE Robotics and Automation Letters. 2020; 5(1), 250-257
Download the full paper here.
For more information, contact us.