NovaSAR-1 Datahub User Guide

The CSIRO NovaSAR-1 National Facility (NNF) Datahub is a web-based system designed to provide bulk dissemination capabilities for NovaSAR-1 image products. The NNF Datahub is based upon the Data Hub System – SciHub – used for the distribution of European Space Agency Copernicus Datasets.

Users must be registered with the Datahub to download imagery. By registering with the NNF Datahub, users agree to the NovaSAR-1 National Facility Datahub Terms and Conditions and the End User Licence Agreement.

Access the NovaSAR-1 National Facility Datahub

Last updated: July 2021

Navigation

1. Access and Registration

2. Graphical User Interface (GUI)

3. Full-Text Search

4. Advanced Search

5. Application Program Interfaces (APIs)

1. Access and Registration

Access the NovaSAR-1 National Facility Datahub

Recommended browsers are:

  • Internet Explorer 10 or above
  • Chrome
  • Firefox
  • Safari
  • Opera

To comply with Export and Sanctions regulations we are unable to distribute NovaSAR-1 data to all countries.

Registration

To register for a CSIRO NovaSAR-1 National Facility Datahub user account follow the below steps:

      • Click the Login icon in the top right of the Home Page.

        CSIRO NovaSAR-1 National Facility Datahub - Login

        CSIRO NovaSAR-1 National Facility Datahub – Login

    • Click Sign Up, this will take you to the Register new account form
    • Complete the form, taking note that:
      • all fields are mandatory
      • the username cannot contain any of the following characters:
        + – & | ! ( ) { } [ ] ^ ” ~ * ? : \ $ % ‘ <blank space>
      • the password must be at least 8 characters long
      • In the current datahub version, the only password characters allowed are alphanumeric and:
        @ ! # $ % ^ & * ) ( + = . , _ –
      • if the chosen username is already taken, a “Username already taken” error message is returned
CSIRO NovaSAR-1 National Facility Datahub - Register new account

CSIRO NovaSAR-1 National Facility Datahub – Register new account

    • If you select the Other option for Domain and/or Usage a new field will appear for you to type in the information
CSIRO NovaSAR-1 National Facility Datahub - Registration Other

CSIRO NovaSAR-1 National Facility Datahub – Registration Other

    • You must read and agree to the NovaSAR-1 National Facility Datahub Terms and Conditions, which outlines the Copyrights, Privacy Policy, Use of Cookies and Service Availability, and also includes the End User Licence Agreement.
    • Click the Register button and you will see the following box pop up unless there are errors that need to be rectified.

      CSIRO NovaSAR-1 National Facility Datahub - Registration SuccessfulLogin

      CSIRO NovaSAR-1 National Facility Datahub – Registration Successful Login

    • To comply with Export and Sanctions regulations we are unable to distribute NovaSAR-1 data to all countries. We will therefore validate your account before you are able to use the CSIRO NNF Datahub. This will be done as soon as possible during working hours. If you have not had your account validated within 2 working days please contact us here

2. Graphical User Interface (GUI)

The GUI provides:

NNF Datahub GUI

NNF Datahub GUI

Map Tool

The map tool provides a geographical search feature for defining an Area of Interest (AOI) in either a box or a polygon. Note there is currently only 1 map layer available so Select Map Layer only displays 1 option.

Users can search in Navigation Mode or Area Mode.

Navigation Mode

  • Left-click to pan around the pan layer
  • Right-click and drag to define a rectangle (box) AOI
  • Right-click a number of times, finally using a double right-click to define a polygon AOI
  • Right-click again to clear your AOIs

Area Mode

  • Right-click to pan around the pan layer
  • Left-click and drag to define a rectangle (box) AOI
  • Left-click a number of times, finally using a double right-click to define a polygon AOI
  • Left-click again to clear your AOIs
NNF Datahub - Box AOI

NNF Datahub – Box AOI

NNF Datahub - Polygon AOI

NNF Datahub – Polygon AOI

The map tool can be combined with the full-text search and the advanced search to further restrict the results.

Upload Shapefile

As well as drawing box and polygon AOIs, the NNF Datahub allows you to upload existing shapefiles.

Only files with extension .shp are supported.

Shapefiles are limited to one record of type POLYGON and must be no larger than 5.00 MB.

Click the Advanced Search icon and expand the Upload Shape File For Geographical Selection section

Click Choose File and navigate to your file then click Upload OR drag and drop your shapefile onto the map.

NNF Datahub - Upload shapefile

NNF Datahub – Upload shapefile

Search Results

The Search results list provides all the products matching the submitted search query.

NNF Datahub - Search Functions

NNF Datahub – Search Functions

Each result consists of:

  • product name
  • image quicklook (when available)
  • product URL (clicking this will initiate the download of the product)
  • the size of the product (this may be displayed as MB or GB)

The footprints of the products matching the search query are displayed on the map. Each footprint gets highlighted when hovering on the corresponding product in the search results list.

NOTE: the map tool only shows the footprints of the products on the current search list page. To widen results modify the page size (25, 50, 75, 100, 125, 150).

Click on Show Product List icon to go back to the product list.

NNF Datahub - Open Product List

NNF Datahub – Open Product List

Save Search

Click on the disk icon to save searches. A green message “User search saved successfully” is displayed in the bottom right-hand corner to confirm.

NNF Datahub - Save Search

NNF Datahub – Save Search

Open the User Profile to retrieve your Saved Searches

NNF Datahub – User Profile

Product Functions

The following options are available for each product displayed in the search list:

  • Select product
  • Zoom to product
  • View Product Details
  • Add Product to Cart
  • Download Product
  • Delete Product
NNF Datahub - Product Functions

NNF Datahub – Product Functions

Click on View Product Details and the following product information can be seen:

  • Product name
  • Product URL
  • Footprint
  • Quicklook
  • Attributes (Summary, Product, Platform, Instrument)
  • Inspector (File Structure)
NNF Datahub - Product Details

NNF Datahub – Product Details

User Profile

Click on User Profile to:

  • See the allowed functionalities: search and download products
  • Cart: allows the user to manage product(s) in the cart
  • Saved searches: allows management of the saved searches
NNF Datahub - User Profile

NNF Datahub – User Profile

Click Edit Profile to access the profile panel.

The profile panel allows you to modify the details provided at registration and the option to change your password.

NNF Datahub - Edit Profile

NNF Datahub – Edit Profile

Cart

The NNF Datahub provides a Cart to which you can add products.

To add products to the Cart, start by performing a query. There are 3 different ways to add products to your Cart:

1) Selecting products individually:

Click on the Add Product to Cart icon under the product result

2) Selecting one or multiple products from the query result panel:

Click on the corresponding checkbox(s) and then click the Trolley icon at the bottom of the search panel to add product(s) to the Cart.

3) Selecting all products from the query result panel:

Check the Select All Items checkbox to select all products that are displayed on the search result panel and then click on the Trolley icon located at the bottom of the search panel.

  • Please note that only the displayed products will be added to the Cart. Should you wish to display more products, edit the number of products displayed on the page using the drop-down menu in the bottom left of the page
  • All products sent to the Cart will be displayed once you open/view your Cart
  • The Cart can be opened either by the Full-Text Bar or from your User Profile

    NNF Datahub - Open Cart

    NNF Datahub – Open Cart

  • Within the Cart there are a number of options available to you.
NNF Datahub - View Cart

NNF Datahub – View Cart

  • Bulk or single downloads are possible
  • The bulk download of the content of the whole Cart can be done through a provided metalink file
    • Click on Download Cart to get the metalink file listing the download URLs of all the products in the cart
    • A pop-up window will prompt for confirmation to save a file named products.meta4
    • A third-party download manager is required to manage this kind of file. A list of the most known download manager can be found here
    • NOTE: For bulk downloads, please do not add more than 100 products to the cart since the metalink file can manage a list of maximum 100 products
  • To perform a single download click on the Download button icon associated with each product
  • Remove single products from the Cart by clicking on the bin icon associated with each product
  • The bin button at the bottom of the search panel removes all the products in the Cart
  • To return to the map click on the green trolley icon in the full-text bar

3. Full-Text Search

Simple Search

Search queries can be performed using either the OpenSearch API or entering a text query in the full-text search bar. The Search Keywords explained on this page can be used with both the API and the full-text search bar. The table below shows some simple text queries that can be typed in the full-text search bar:

Text Query Description
blank field returns all products in the archive
* returns all products in the archive
SLC returns all SLC products
GRD returns all GRD products
ASCENDING returns all products of the ascending passes
HH returns all products containing a measurement dataset in HH polarisation
VH returns all products containing a measurement dataset in VH polarisation
HH+HV returns all products containing measurement datasets in HH and HV polarisations

Wildcards and Operators

Wildcards and Operators can be used for restricting search queries. Permitted wildcards and operators are shown in the table below:

Wildcard
*QUERY* any sequence of zero or more characters
? any one character
Operator
AND Narrow search and retrieve records containing all of the words it separates.
OR Broaden search and retrieve records containing any of the words it separates.
NOT Narrow search and retrieve records that do not contain the term following it.
( ) Group words or phrases when combining Boolean phrases and to show the order in which relationships should be considered.

Keywords

It is also possible to make full-text queries using search keywords.

The syntax is the following:

<keyword>:<values>

Depending on the keyword, the value(s) can be specified as a single value or range of values.

Search keywords can be combined with each other using Operators.

The following table contains the list of the keywords and their corresponding usage.

Search Keyword Syntax and Examples
platformname: Used to perform a search based on the Satellite Platform name.

Syntax:

platformname: <platform name>

Examples:

  • platformname:NovaSAR-1
beginposition:
endposition:
ingestiondate:		 These <keyword> are used to perform a time interval search based on:

  • beginposition: Sensing Start Time (or Validity Start Time for Aux Data)
  • endposition: Sensing Stop Time (or Validity End Time for Aux Data)
  • ingestiondate: the time when the product is indexed in the databases. The archive products in the database were all added in March and April 2021, for products added since then, the ingestion date is usually within a day or 2 of acquisition.

The general form to be used is:

  • <keyword>:[<timestamp> TO <timestamp>]
    • where <timestamp> can be expressed in one of the following formats:
      • yyyy-MM-ddThh:mm:ss.SSSZ (ISO8601 format)
      • NOW
      • NOW-<n>MINUTE(S)
      • NOW-<n>HOUR(S)
      • NOW-<n>DAY(S)
      • NOW-<n>MONTH(S)

      with n = 1, 2, 3, …

      Examples:

      • beginposition:[2021-01-01T00:00:00.000Z TO NOW]
      • beginposition:[2021-01-01T00:00:00.000Z TO 2021-02-01T00:00:00.000Z]
      • beginposition:[NOW-2DAYS TO NOW]
      • endposition:[2021-01-01T00:00:00.000Z TO NOW]
      • endposition:[2021-01-01T00:00:00.000Z TO 2021-02-01T00:00:00.000Z]
      • endposition:[NOW-1HOUR TO NOW]
      • ingestiondate:[2021-01-01T00:00:00.000Z TO NOW]
      • ingestiondate:[2021-01-01T00:00:00.000Z TO 2021-02-01T00:00:00.000Z]
      • ingestiondate:[NOW-30MINUTES TO NOW]
antennapointing: The direction of the orbit in the product

Possible values for <antennapointing> are: Left, Right

Examples:

  • antennapointing:left
filename: Used to perform a search based on the product filename.

Syntax:

filename:<filename>

where <filename> can be either expressed as full product file name according to product naming convention or as partial product file name using wildcards.

Examples:

  • filename:*22149*
  • filename:NovaSAR_01_22149_scd_32_210422_142356_HH_HV
  • filename:*grd*210404*

The complete explanation of the product naming convention is available at the following link:
footprint: A geographical search of the products whose footprint intersects or is included in a specific geographic type.

Syntax:

footprint:"intersects(<geographic type>)"

The <geographic type> value can be expressed as a polygon or as a point, according to the syntax described below:
GEOGRAPHIC TYPE:

POLYGON:

<geographic type>=POLYGON((P1Lon P1Lat, P2Lon P2Lat, …, PnLon PnLat, P1Lon P1Lat))

where P1Lon and P1Lat are the Longitude and Latitude coordinates of the first point of the polygon in decimal degrees (DDD) format (e.g. 2.17403, 41.40338) and so on.
The coordinates of the last point of the polygon must coincide with the coordinates of the first point of the polygon.
There is no fixed limit for the maximum number of points to be used to define the Area of Interest (AOI). The limit is related to the maximum number of characters for the GUI search (i.e 8000). The maximum footprint extent possible is defined by the Mercator projection, i.e. LAT [-180, + 180] and LONG [-85.05, +85.05].

Example:

The polygon in the example is in central Australia:

  • footprint:"Intersects(POLYGON((133.59 -25.10, 133.93 -25.98, 134.29 -25.17, 134.24 -25.55, 133.51 -25.49, 133.59 -25.10)))"

POINT:

<geographic type>=Lat, Lon

where the Latitude (Lat) and Longitude (Lon) values are expressed in decimal degrees (DDD) format (e.g. -40.40338, 130.17403 ).

Examples:

  • Sydney – footprint:"intersects(-33.8688, 151.2093)"
  • Melbourne – footprint:"intersects(-37.8136, 144.9631)"
  • Brisbane – footprint:"intersects(-27.4705, 153.0260)"
  • Perth – footprint:"intersects(-31.9523, 115.8613)"
  • Canberra – footprint:"intersects(-35.2809, 149.1300)"
  • Adelaide – footprint:"intersects(-34.9285, 138.6007)"
  • Hobart – footprint:"intersects(-42.8826, 147.3257)"
  • Darwin – footprint:"intersects(-12.46370, 130.8444)"
passdirection: The direction of the orbit in the product

Possible values are: Ascending, Descending

Examples:

  • passdirection:Ascending
polarisation: Search the valid polarisations for NovaSAR-1. Note: Searching HH will return all images that contain HH so you may want to combine this with another search function.

Possible values for <polarisation> are listed here

Syntax

polarisation:<polarisation>

Examples

  • polarisation:HH
  • polarisation:VV HH HV
producttype: Used to perform a search based on the product type.
Syntax:

producttype:<producttype>

Possible values for <producttype> are: SLC, GRD, SRD, SCD

Examples

  • producttype:SLC
  • producttype:GRD
operationalmode: Valid operational modes for the NovaSAR-1 instrument.

Possible values for <operationalmode> are listed here

Syntax

operationalmode:<operationalmode>

Examples:

  • operationalmode:Maritime
modeid: modeid:

Search all valid mode IDs for NovaSAR-1. Useful if you only want images with the same viewing parameters.

Possible values are: ADD LINK

Syntax:

modeid:<modeid>

Examples:

  • modeid:2001

4. Advanced Search

The advanced search panel is shown by clicking on the Advanced Search icon

NNF Datahub - Advanced Search

NNF Datahub – Advanced Search

NNF Datahub - Advanced Search Panel

NNF Datahub – Advanced Search Panel

It consists of different search fields. Each search field allows you to perform specific queries on the associated parameter. Each search field can be used singularly or in combination with the others search fields, the full-text search bar and the geographic map tool.

Search Field Usage
Sort by This field allows sorting of the search results by:

  • Ingestion date (represents the time when the product is indexed in the datahub service databases. Time is in UTC)
  • Sensing Date (represents the time when the product has been taken. Time is in UTC)
Order by This field allows you to sort the search results in Ascending or Descending order.
Sensing Period This search field is composed of two date entries. Dates have to be inserted for both entries. “Pick up date” calendars allow date selection. The query returns all the products whose sensing dates and times are included in the defined period. In particular, it returns all the products that respond to both of the following criteria:

  • sensing start time equal or greater than 00:00:00 (hh:mm:ss) of the first selected date
  • sensing stop time equal or less than 23:59:59 (hh:mm:ss) of the second selected date

The Sensing Time corresponds to the time of the satellite on-board acquisition and it is stamped for each line of the acquired image scene. The sensing start and stop times of a product correspond to the time of the satellite on-board acquisition of respectively the first and last line of the image in the product. Click on Clear to remove the search entry. Click on Today to select the current date. Click on Close to return to the Advanced search menu.
Ingestion Period This search field is composed of two date entries as with Sensing Period.
The query returns all the products whose ingestion dates and times to the NNF datahub databases are included in the defined period.
In particular, it returns all the products that respond to both of the following criteria:

  • Publication time equal or greater than 00:00:00 (hh:mm:ss) of the first selected date
  • Publication time equal or less than 23:59:59 (hh:mm:ss) of the second selected date.

Please note that the ingestion date does not correspond to the generation date of the product processed at the ground segment. The ingestion date represents the time when the product is indexed in the datahub service databases. The archive products in the database were all added in March and April 2021, for products added since then, the ingestion date is usually within a day or 2 of acquisition. Time is in UTC.
Mission NovaSAR
Product Type Accepted entries are:

  • SLC (Single Look Complex, Slant Range)
  • GRD (Multi Look Detected, Ground Range)
  • SRD (Multi Look Detected, Slant Range)
  • SCD (Multi Look Detected, Ground Range)
Antenna Pointing Accepted entries are:

  • Left
  • Right
Operational Mode Accepted entries are:

  • Maritime
  • 6m_Stripmap_HH
  • 6m_Stripmap_VV
  • 20m_ScanSAR_HH
  • 20m_ScanSAR_VV
  • 30m_ScanSAR_HH
  • 30m_ScanSAR_VV
  • 20m_2Pol_ScanSAR_HHVV
  • 30m_3Pol_ScanSAR_VVHHHV
  • 40m_ScanSAR_HHHV
  • 45m_ScanSAR_HHHV
  • 33m_ScanSAR_HH
  • 35m_3Pol_ScanSAR_VVHHHV
  • 50m_ScanSAR_HHHV
  • 40m_ScanSAR_HV
  • 20m_ScanSAR_HHHV
Pass direction Accepted entries are:

  • ASCENDING
  • DESCENDING
Polarisation Accepted entries are:

  • HH
  • VV
  • HH VV
  • VV HH HV
  • HH HV
  • HV

5. Application Program Interfaces (APIs) 

The NNF Datahub exposes two dedicated Application Program Interfaces (API): OData and OpenSearch. Both can be used for searching/filtering the archive for specific products, and each has some advantages over the other. 

  • The SciHub guides linked below provide good background information and additional examples
  • Some usage examples follow (Note that only formatted URL examples are provided)
  • It’s possible to use a browser directly using a correctly formatted URL, and the browser should prompt for a user/pass combination (unless this has been previously entered)
  • It is also possible to use command-line tools (eg. curl) or write your own code (eg. Python) to use the APIs 
  • When using the command-line under Linux/macOS, the URL may need to be enclosed in quotes, and some special characters may need to be escaped in order to be interpreted correctly
  • NOTE: All API calls require authentication

OData

SciHub OData API Guide 

Allows searching by name, downloading data. 

Search Parameters URL example
# list of products  http://data.novasar.csiro.au/odata/v1/Products?$orderby=IngestionDate%20desc%20&$top=10&$format=json
# just pick some fields  http://data.novasar.csiro.au/odata/v1/Products?$orderby=IngestionDate%20desc%20&$top=10&$select=Id,Name&$format=text/csv
# download product  http://data.novasar.csiro.au/odata/v1/Products(‘2526383a-5053-4957-9bca-a859cebdf9c1’)/$value
# search by name  http://data.novasar.csiro.au/odata/v1/Products?$filter=Name%20eq%20%27NovaSAR_01_10097_scd_200212_145523_HH%27&$select=Name,Id&$format=text/csv
# get a count of all products (again using ‘curl’ and ‘jq’) curl -s -u user:pass “http://data.novasar.csiro.au/odata/v1/Products?\$inlinecount=allpages&\$format=json” |jq ‘.d.__count’ 
# need to loop through to get > 100 prods (DHuS returns a maximum of 100 results) 

# $top and $skip (top max = 100) 

 http://data.novasar.csiro.au/odata/v1/Products?$orderby=IngestionDate%20desc%20&$skip=100&$top=10&$format=json

OpenSearch 

SciHub OpenSearch API Guide  

Allows searching by polygon, date/time ranges. 

Search Parameters URL example
# search by ingestion date  https://data.novasar.csiro.au/search?q=(ingestiondate:%5bNOW-2DAYS%20TO%20NOW%5d)&orderby=ingestiondate%20desc&format=json
# search by polygon  https://data.novasar.csiro.au/search?q=(footprint:%22Intersects(POLYGON((148.18359374999997%20-36.315125147480494,150.11718749999997%20-36.315125147480494,150.11718749999997%20-34.30714385628803,148.18359374999997%20-34.30714385628803,148.18359374999997%20-36.315125147480494)))%22)
# all in one (using ‘curl’ and piping output to the json query tool ‘jq’)  curl -s -u user:pass ‘https://data.novasar.csiro.au/search?q=(ingestiondate:%5bNOW-3DAYS%20TO%20NOW%5d%20AND%20footprint:%22Intersects(POLYGON((148.18359374999997%20-36.315125147480494,150.11718749999997%20-36.315125147480494,150.11718749999997%20-34.30714385628803,148.18359374999997%20-34.30714385628803,148.18359374999997%20-36.315125147480494)))%22)&orderby:ingestiondate%20desc&format=json‘ |jq ‘.feed.entry[] | “\(.title),\(.id)” ‘ 

See the SciHub Batch Scripting Guide for information on how to combine the OData and OpenSearch URI’s using cURL and Wget.