parsl.executors.base.ParslExecutor

class parsl.executors.base.ParslExecutor(*, monitoring_messages: Queue[TaggedMonitoringMessage] | None = None, run_dir: str = '.', run_id: str | None = None)

Executors are abstractions that represent available compute resources to which you could submit arbitrary App tasks.

This is an abstract base class that only enforces concrete implementations of functionality by the child classes.

Can be used as a context manager. On exit, calls self.shutdown() with no arguments and re-raises any thrown exception.

In addition to the listed methods, a ParslExecutor instance must always have a member field:

label: str - a human readable label for the executor, unique

with respect to other executors.

Per-executor monitoring behaviour can be influenced by exposing:

radio_mode: str - a string describing which radio mode should be used to

send task resource data back to the submit side.

An executor may optionally expose:

storage_access: List[parsl.data_provider.staging.Staging] - a list of staging

providers that will be used for file staging. In the absence of this attribute, or if this attribute is None, then a default value of parsl.data_provider.staging.default_staging will be used by the staging code.

Typechecker note: Ideally storage_access would be declared on executor __init__ methods as List[Staging] - however, lists are by default invariant, not co-variant, and it looks like @typeguard cannot be persuaded otherwise. So if you’re implementing an executor and want to @typeguard the constructor, you’ll have to use List[Any] here.

The DataFlowKernel will set this attribute before calling .start(), if monitoring is enabled:

monitoring_messages: Optional[Queue[TaggedMonitoringMessage]] - an executor

can send messages to the monitoring hub by putting them into this queue.

__init__(*, monitoring_messages: Queue[TaggedMonitoringMessage] | None = None, run_dir: str = '.', run_id: str | None = None)

Methods

__init__(*[, monitoring_messages, run_dir, ...])
monitor_resources()	Should resource monitoring happen for tasks on running on this executor?
shutdown()	Shutdown the executor.
start()	Start the executor.
submit(func, resource_specification, *args, ...)	Submit.

Attributes

label
radio_mode
run_dir	Path to the run directory.
run_id	UUID for the enclosing DFK.

label: str = 'undefined'

monitor_resources() → bool

Should resource monitoring happen for tasks on running on this executor?

Parsl resource monitoring conflicts with execution styles which use threads, and can deadlock while running.

This function allows resource monitoring to be disabled per executor implementation.

radio_mode: str = 'udp'

property run_dir: str

Path to the run directory.

property run_id: str | None

UUID for the enclosing DFK.

abstract shutdown() → None

Shutdown the executor.

This includes all attached resources such as workers and controllers.

abstract start() → None

Start the executor.

Any spin-up operations (for example: starting thread pools) should be performed here.

abstract submit(func: Callable, resource_specification: Dict[str, Any], *args: Any, **kwargs: Any) → Future

Submit.

The executor can optionally set a parsl_executor_task_id attribute on the Future that it returns, and in that case, parsl will log a relationship between the executor’s task ID and parsl level try/task IDs.