Systems

Software Engineering

Ocean, atmosphere and littoral models are sophisticated pieces of software that implement numerical schemes to simulate environmental physics. They are sensitive to their input forcing data and the parameter sets with which they run, and the resources (eg storage volume and CPU cores) that are available.  Making them run reliably and repeatedly requires careful orchestration and precise attention to detail.

The software engineering team is responsible for ensuring the models are run in an environment that is sufficiently well-resourced and provides access to the necessary input data sets.  In addition, there is a web-based user interface that enables appropriately credentialed users to define spatial domains and timeframes, select from a range of forcing data choices, and then to access the output datasets and log files.  An online user guide is also an important component of a production system. The model codes are frequently improved as part of the research process.  Wherever these changes affect the choices users make, the input data demands, or the resource environment, the orchestration system and/or user interface and/or user guide need to evolve in step.

To support a high availability production software system, many deployed instances are required. To provide a resilient service multiple (alternate) compute platforms are used, while other instances are needed for testing of issues that arise during operations, and for developing new versions.  To manage this workload, the engineering team has adopted a progressively more modular approach to the system architecture. In turn this has permitted exploitation of the containerisation paradigm to provide rapid deployment and platform agility.  Deeply embedded architectural choices such as these create the operational efficiencies necessary to allow more engineering time to be spent on system improvements.

Data management

The models consume and produce large volumes of data.  The forcing data streams are diverse (ocean model output, numerical weather predictions, satellite and in situ observations) and extensive (global, all times).  Correctly selecting and accessing these data is key to successfully running the Bluelink models.  Furthermore, timely acquisition of these data is essential if they are to support forecasts in the near future.  Active curation of all these data streams, from acquisition through checking completeness and maintaining a catalogue, is therefore a fundamental activity supporting the generation of environmental predictions within a useful time frame.

The ROAM data management system is designed to manage all these demands in a way that is essentially invisible to the user.  It provides a service to the model run orchestration system that enables querying of the available forcing data sets, and an interface that selects the requested data and packages it in the form required by the models. A significant software and storage infrastructure underpins the service. For many years it was a unique system ahead of its time.  The immense recent growth in web catalogue, metadata, and data transfer protocols, standards and software tools has opened an opportunity for modernisation.  This will involve an evolution of the architecture to replace bespoke code with off-the shelf packages, and will also take advantage of the gains afforded by containerisation.