Paper: Benchmarking simulated robotic manipulation through a real world dataset
We present a benchmark to facilitate simulated manipulation; an attempt to overcome the obstacles of physical benchmarks through the distribution of a real world, ground truth dataset.
Users are given various simulated manipulation tasks with assigned protocols having the objective of replicating the real world results of a recorded dataset.
The benchmark comprises of a range of metrics used to characterise the successes of submitted environments whilst providing insight into their deficiencies.
We apply our benchmark to two simulation environments, i.e., PyBullet and V-Rep, and publish the results.
All materials required to benchmark an environment, including protocols and the dataset, can be found at the benchmarks’ website here.
Collins, Jack; McVicar, Jessie; Wedlock, David; Brown, Ross; Howard, David; Leitner, Jurgen. Benchmarking simulated robotic manipulation through a real world dataset. IEEE Robotics and Automation Letters. 2020; 5(1), 250-257
Download the full paper here.
For more information, contact us.