parsl.executors.radical.RadicalPilotExecutor

class parsl.executors.radical.RadicalPilotExecutor(resource: str, label: str = 'RPEX', bulk_mode: bool = False, working_dir: str | None = None, rpex_cfg: ResourceConfig | None = None, **rpex_pilot_kwargs)

Executor is designed for executing heterogeneous tasks

in terms of type/resource.

The RadicalPilotExecutor system has the following main components:

1. “start”: Create and start the RADICAL-Pilot runtime components rp.Session,

   rp.PilotManager and rp.TaskManager.

2. “translate”: Unwrap, identify, and parse Parsl apps into rp.TaskDescription.

3. “submit”: Submit Parsl apps to rp.TaskManager.

4. “shutdown”: Shut down the RADICAL-Pilot runtime and all associated components.

Here is a diagram

----------------------------------------------------------------------------
         Parsl Data Flow Kernel        |   Task Translator |  rp.TaskManager
---------------------------------------|-------------------|----------------
                                       |                   |
-> Dep. check ------> Parsl_tasks{} <--+--> Parsl Task     |
 Data management          +dfk.submit  |        |          |
                                       |        v          |
                                       |    RP Task(s) ->  | submit(task)
----------------------------------------------------------------------------

The RadicalPilotExecutor creates a rp.Session, rp.TaskManager, and rp.PilotManager. The executor receives the Parsl apps from the DFK and translates these apps (in-memory) into rp.TaskDescription object to be passed to the rp.TaskManager. This executor has two submission mechanisms:

1. Default_mode: where the executor submits the tasks directly to RADICAL-Pilot.

2. Bulk_mode: where the executor accumulates N tasks (functions and executables) and submit them.

Parameters:

• rpex_cfg – a dataclass specifying resource configuration. Default is ResourceConfig instance.

• label (str) – Label for this executor instance. Default is “RPEX”.

• bulk_mode (bool) – Enable bulk mode submission and execution. Default is False (stream).

• resource (Optional[str]) – The resource name of the targeted HPC machine or cluster. Default is local.localhost (user local machine).

• runtime (int) – The maximum runtime for the entire job in minutes. Default is 30.

• working_dir (str) – The working dir to be used by the executor.

• rpex_pilot_kwargs (Dict of kwargs that are passed directly to the rp.PilotDescription object.) –

• information (For more) –

__init__(resource: str, label: str = 'RPEX', bulk_mode: bool = False, working_dir: str | None = None, rpex_cfg: ResourceConfig | None = None, **rpex_pilot_kwargs)

Methods

__init__(resource[, label, bulk_mode, ...])
monitor_resources()	Should resource monitoring happen for tasks on running on this executor?
shutdown([hub, targets, block])	Shutdown the executor, including all RADICAL-Pilot components.
start()	Create the Pilot component and pass it.
submit(func, resource_specification, *args, ...)	Submits tasks in stream mode or bulks (bulk mode) to RADICAL-Pilot rp.TaskManager.
task_state_cb(task, state)	Update the state of Parsl Future apps Based on RP task state callbacks.
task_translate(tid, func, ...)	Convert Parsl function to RADICAL-Pilot rp.TaskDescription
unwrap(func, args)	Unwrap a Parsl app and its args for further processing.

Attributes

hub_address	Address to the Hub for monitoring.
hub_zmq_port	Port to the Hub for monitoring.
label
radio_mode
run_dir	Path to the run directory.
run_id	UUID for the enclosing DFK.
submit_monitoring_radio	Local radio for sending monitoring messages

shutdown(hub=True, targets='all', block=False)

Shutdown the executor, including all RADICAL-Pilot components.

start()

Create the Pilot component and pass it.

submit(func, resource_specification, *args, **kwargs)

Submits tasks in stream mode or bulks (bulk mode) to RADICAL-Pilot rp.TaskManager.

task_state_cb(task, state)

Update the state of Parsl Future apps Based on RP task state callbacks.

task_translate(tid, func, parsl_resource_specification, args, kwargs)

Convert Parsl function to RADICAL-Pilot rp.TaskDescription

unwrap(func, args)

Unwrap a Parsl app and its args for further processing.

Parameters:

• func (callable) – The function to be unwrapped.

• args (tuple) – The arguments associated with the function.

Returns:

A tuple containing the unwrapped function, adjusted arguments, and task type information.

Return type:

tuple