Senaps-LAND and data staging

Transforming stored and real-time raw data into product-ready information for agriculture and land management applications

What is the data ‘problem’?

There is tremendous growth in the data we can access from many sources including satellites, machinery, networked in-field sensors, Internet of Things technologies, and mobile phone apps. These data are heterogeneous in their meaning, their format and the communication systems that carry them. Collection of the data is often distributed across sizeable areas, involves third parties and multi-tiered device capabilities, and carries an ongoing risk of hardware failures.

If these rich monitoring data are going to be used to inform decisions, they must be captured, organised, and transformed using coherent and reliable methods and in ways that ensure that they can be trusted. To reach the Digiscape goal of building a variety of different agricultural and land management decision support tools efficiently, we needed methods to combine these monitoring data with model-based forecasts and other analyses and then to provide them to decision-makers.

What science challenges / questions did we address?

Our 3-year research effort, which ended in June 2020, helped tackle the following science challenges:

  • How can sensor networks convey more resilient information despite environmental effects (e.g. storms) and system limitations (e.g. transient access failure, battery life)?
  • How to optimally deploy the components of these data staging services in a distributed manner across the sizeable sets of third parties and multi-tiered devices involved?
  • How to design dynamic mechanisms allowing these components to automatically adjust to changes in requirements or contexts (e.g. hardware failure, environment-induced faults)?

This research produced several novel methods and algorithms, which together push further the limits of what wireless networked sensors can achieve while collecting and transmitting field data in land sector applications.

Take for example a scenario where sensors have fixed battery life. One of our novel algorithms computes the best coding rate for transmitting data to minimise the overall energy usage by the entire set of sensors. This in turn allows the entire set of sensors to function for a longer period of time before going flat.

Another one of our novel methods finds the best coding rates for all sensors to achieve the same knowledge about their environment with minimal transmissions, again allowing all sensors to last longer in the field. In addition, this state of shared common knowledge between sensors allows them to dynamically adapt their behaviour. For example, when measuring chemicals in an estuary, downstream sensors can increase their sampling rate if they know that upstream sensors have detected high reading fluctuations, thereby being able to capture more information about a potential ongoing input event in the estuary.

The research contributions of this project have been published in peer-reviewed high-ranked scientific venues. We successfully completed some initial trials of one of these algorithms on real-world data in the context of canopy temperature measurements in tomato fields, and are looking for further applications within the land sector.

Senaps-LAND: our solution to the data ‘problem’

In concert with our research effort into sensor networks, we built Senaps-LAND, a data staging service that is specifically designed for land sector applications. Senaps-LAND transforms raw data into “product-ready” information suitable for Digiscape’s services and applications, and allows stored or real-time sensor data to be combined with the predictive models required by each of the Digiscape applications. We used CSIRO’s already existing Senaps technology to construct Senaps-LAND.

Senaps-LAND:

  • ensures that the knowledge and information derived from staged data is trusted and traceable, through quality assurance and provenance services
  • guarantees data integrity/security, and the privacy of its sources
  • embeds existing predictive models (e.g. APSIM) into larger decision support work flows that depend on sensor data
  • allows third-party services and applications to have easier access to richer data and better efficiency by re-using common building blocks across application domains (i.e. “develop once, use many times”)
  • unites disparate environmental information into common data models
  • provides high standards of software quality.

Senaps-LAND is up and running and not only available for Digiscape projects but also businesses delivering near real-time data and complex analytics requiring the right technology to do it securely, efficiently and at scale.

Where can I find out more about Senaps-LAND?

Find out more about Senaps and Senaps-LAND and the customers the technology is helping on the Senaps website: products.csiro.au/senaps/.

The work is led by Dr Chris Sharman and Dr Thierry Rakotoarivelo.

  • Thierry's research focuses on data privacy and information security, and their application to federated systems. Thierry led the Digiscape's data staging research project.