Workspace 6.21.5
Workspace parallelism - an overview and some caveats
[ Previous: Basic SQL Database ] [ Up: Introductory Tutorials ] [ Next: Running workflows in parallel on a local machine ]

An overview

Workspace can parallelise and distribute execution of a workflow.

Workspace workflows can be:

  1. Executed in parallel on the local machine in shared memory (the default, referred to as "Shared memory"). For details on this see tutorial Running workflows in parallel on a local machine.
  2. Executed out-of-process on a remote machine over a TCP/IP based socket connection via the Workspace-server service (referred to as "Workspace-server"). For details on this see tutorial Distributing workload to other (non-cluster) remote machines with the Workspace-Server service.
  3. Executed out-of-process on a remote PBS/SLURM based cluster over SSH (referred to as "Clustered"). For details on this see tutorial Distributing work to a compute cluster.

Depending on your situation, some of the above configurations may not be appropriate.

Below is a table summarising the applicability of the above mentioned parallelism scenarios based on your usage scenario:

Scenario Shared Memory Out of process Workspace-server Clustered
Your operation code is thread-safe (including any referenced third-party libraries) Safe Safe Safe Safe
Your operation code is NOT thread-safe (or any referenced third-party libraries are NOT thread safe) Not Safe Safe Safe Safe

If your Operation code (or code your Operation references) is not thread safe then you should opt to use a parallelism mode other than the default shared-memory mode. Failure to do so may result in the workflow failing to execute when parallelism is employed.

NB. In a future release this detail will be abstracted away by having an Operation author explicitly declare to the Workspace whether or not an Operation is thread-safe. Under such support an appropriate parallelism mode would be automatically selected by the Workspace's internal scheduling system.


[ Previous: Basic SQL Database ] [ Up: Introductory Tutorials ] [ Next: Running workflows in parallel on a local machine ]