Benchmarking real-world manipulation tasks – call for participation
Expanding from his work in Quantifying the Reality Gap in Robotic Manipulation Tasks, Jack Collins, one of our PhD students, along with his supervisors David Howard, Juxi Leitner and Ross Brown, have just released a portal for researchers to validate their simulation environments using a benchmark comprised of real-world manipulation tasks.
The goal of the project is to gain further insight into physics simulation, particularly for robotic manipulation, and Mr Collins and his team are inviting researchers, developers or anyone with interest to visit their portal, access the available datasets, run simulations and submit their results for benchmarking.
According to Mr Collins, “there are currently many rigid-body physics engines available as both open source and proprietary software for researchers to use, the problem is that we are unable to rank and compare how realistic they are. Our benchmark and dataset work towards solving this problem so when applying a physics simulator to your problem, you are able to apply the best one”.
Mr Collins explains that “their dataset is unique in that we have a robot performing tasks and we record as much information from the robot and the environment including positions, torques and contacts moments and forces”.
The state-of-the-art Motion Capture System at CSIRO’s Data61 Robotics Innovation Centre provided an accurate and sophisticated method for comparing simulation to the real world.
“The metrics and measures used in this project were chosen as they numerically compare important simulation characteristics like velocities, accelerations, maximum torques and pose errors”, says Mr Collins.
By participating in the benchmarking, developers or users will be able to quantify how realistic their setup is as a path towards validation and improvement.
While the code and the data will remain available for approximately 6 years, users can submit and publicly rank results for a period of approximately two years.
To learn more, visit our Manipulation Benchmark portal or contact Jack Collins at Jack.Collins@data61.csiro.au