The Information and Data Centre (IDC) uses a variety of tools and software to develop applications for the distribution of data to both the public and CSIRO scientific staff. Discovery and delivery is typically web-based, and we promote the ability of users to self discover and download data where possible. Our team also assists with managing more complex data queries, and provides a range of visualisations of our data products.
We support a number of applications that source data from many locations and present it seamlessly to end users. Some of our major applications (e.g. Data Trawler) are written in Lucee. This open-source language is targeted at server-side web development and can aggregate multiple data sources (e.g. Files, LDAP and FTP servers and a wide range of databases).
The IDC team extract, transform and load raw data into our enterprise databases, and we make extensive use of modern web frameworks. We are able to provide responsive and interactive systems with dynamic mapping, and complex data visualisations (e.g. plots of CTD profiles). The ability to visualize and subset large datasets prior to download is a major benefit for our clients.
The IDC team also have expertise in Oracle Application Express (APEX), a web development environment tightly integrated with the Oracle database system, enabling data driven web applications to be rapidly developed and deployed.
The Data Centre also uses Java, R and Python to extract, transform and load data into Oracle databases and to also construct plots (e.g. CTD profiles).
A variety of web services are supported to assist with the publication and retrieval of metadata and data. Open Geospatial Consortium (OGC) compliant map services are provided through the NCMI Geoserver. This serves over 1000 map layers from our Data Warehouse as well as serving base layers for the production of maps. This provides context within our own applications and these protocols (WFS, WMS) can be used directly by desktop GIS application (e.g. ArcGIS, QGIS).
The Marlin metadata system is based on GeoNetwork, which also supports the latest OGC and ISO standards. These standards enable us to provide automated data transfers to our national and international partners (e.g. AODN portal, OBIS). The IDC team can assist with the query and retrieval of records using these standards.
- IDC Web development – pdf with additional details