Human Activity Recognition and Behaviour Analysis Through Synthetic Environments

August 3rd, 2020

Overview

Artificial intelligence is impacting the world in many ways. The capability of machine learning algorithms for AI profoundly relies upon the data. Advanced AI/ML models are restricted by both the quality and the amount of data. With the help of simulation, researchers can generate large-scale synthetic training data sets and develop innovative models before transferring the learned skills to reality.

Synthetic data generation

Using simulation, computer graphics, and physics-based modelling, we are creating large datasets of synthetic videos that can be used for training and testing machine learning methods.

This augmented data can be used to improve action recognition models, particularly through robustness to factors such as camera viewpoints, actor appearance, background and intra-class variation (e.g. the same action looks very different when performed by different people in different environments).

Rendering synthetic action using computer graphics engines requires 3D human motion to drive 3D human models. This requires the capture of the 3D motion of people undertaking several actions of interest as well as building statistical libraries of realistic action and behaviour.

We expect that augmentation with simulated data will boost the performance of action recognition models on unseen environments and for complex behaviours of interest that are difficult to simulate in open environments. It will also allow to test hypothesis (e.g. camera break down or camera placement), as well as study machine learning limitations such as bias and overfitting.

Significance

This technology is a significant step forward towards the development of AI-based frameworks to augment real videos with large-scale rendered synthetic videos to improve robustness for human action recognition.

By automating the building of new scenarios in a simulated environment, we can assess actions in a real location that is complex to access due to ethical, confidential and other physical restrictions, such as airports, shopping centres and sensitive locations (e.g. government buildings).

Collaborative platform

Synthetic data generation is currently being investigated for several projects, including a collaboration between CSIRO, Department of Defence and the Queensland University of Technology.