parsl.executors.radical.RadicalPilotExecutor

class parsl.executors.radical.RadicalPilotExecutor(resource: str, label: str = 'RPEX', bulk_mode: bool = False, working_dir: str | None = None, rpex_cfg: ResourceConfig | None = None, **rpex_pilot_kwargs)[source]
Executor is designed for executing heterogeneous tasks

in terms of type/resource.

The RadicalPilotExecutor system has the following components:

  1. “start” :creating the RADICAL-executor session and pilot.

  2. “translate”:unwrap/identify/ out of parsl task and construct RP task.

  3. “submit” :translating and submitting Parsl tasks to Radical Pilot.

  4. “shut_down”:shutting down the RADICAL-executor components.

Here is a diagram

----------------------------------------------------------------------------
         Parsl Data Flow Kernel        |   Task Translator |  rp.TaskManager
---------------------------------------|-------------------|----------------
                                       |                   |
-> Dep. check ------> Parsl_tasks{} <--+--> Parsl Task     |
 Data management          +dfk.submit  |        |          |
                                       |        v          |
                                       |    RP Task(s) ->  | submit(task)
----------------------------------------------------------------------------

The RadicalPilotExecutor creates a rp.Session, rp.TaskManager, and rp.PilotManager. The executor receives the parsl apps from the DFK and translates these apps (in-memory) into rp.TaskDescription object to be passed to the rp.TaskManager. This executor has two submission mechanisms:

  1. Default_mode: where the executor submits the tasks directly to RADICAL-Pilot.

  2. Bulk_mode: where the executor accumulates N tasks (functions and executables) and submit them.

Parameters:
  • rpex_cfg – a dataclass specifying resource configuration. Default is ResourceConfig instance.

  • label (str) – Label for this executor instance. Default is “RPEX”.

  • bulk_mode (bool) – Enable bulk mode submission and execution. Default is False (stream).

  • resource (Optional[str]) – The resource name of the targeted HPC machine or cluster. Default is local.localhost (user local machine).

  • runtime (int) – The maximum runtime for the entire job in minutes. Default is 30.

  • working_dir (str) – The working dir to be used by the executor.

  • rpex_pilot_kwargs (Dict of kwargs that are passed directly to the rp.PilotDescription object.) –

  • information (For more) –

__init__(resource: str, label: str = 'RPEX', bulk_mode: bool = False, working_dir: str | None = None, rpex_cfg: ResourceConfig | None = None, **rpex_pilot_kwargs)[source]

Methods

__init__(resource[, label, bulk_mode, ...])

create_monitoring_info(status)

Create a monitoring message for each block based on the poll status.

monitor_resources()

Should resource monitoring happen for tasks on running on this executor?

shutdown([hub, targets, block])

Shutdown the executor, including all RADICAL-Pilot components.

start()

Create the Pilot component and pass it.

submit(func, resource_specification, *args, ...)

Submits tasks in stream mode or bulks (bulk mode) to RADICAL-Pilot rp.TaskManager.

task_state_cb(task, state)

Update the state of Parsl Future tasks Based on RP task state callbacks.

task_translate(tid, func, ...)

Convert parsl function to RADICAL-Pilot rp.TaskDescription

unwrap(func, args)

Unwrap a parsl app and its args for further processing.

Attributes

hub_address

Address to the Hub for monitoring.

hub_port

Port to the Hub for monitoring.

label

radio_mode

run_dir

Path to the run directory.

shutdown(hub=True, targets='all', block=False)[source]

Shutdown the executor, including all RADICAL-Pilot components.

start()[source]

Create the Pilot component and pass it.

submit(func, resource_specification, *args, **kwargs)[source]

Submits tasks in stream mode or bulks (bulk mode) to RADICAL-Pilot rp.TaskManager.

task_state_cb(task, state)[source]

Update the state of Parsl Future tasks Based on RP task state callbacks.

task_translate(tid, func, parsl_resource_specification, args, kwargs)[source]

Convert parsl function to RADICAL-Pilot rp.TaskDescription

unwrap(func, args)[source]

Unwrap a parsl app and its args for further processing.

Parameters:
  • func (callable) – The function to be unwrapped.

  • args (tuple) – The arguments associated with the function.

Returns:

A tuple containing the unwrapped function, adjusted arguments, and task type information.

Return type:

tuple