parsl.executors.ThreadPoolExecutor

class parsl.executors.ThreadPoolExecutor(label: str = 'threads', max_threads: int = 2, thread_name_prefix: str = '', storage_access: List[Any] = None, working_dir: Optional[str] = None, managed: bool = True)[source]

A thread-based executor.

Parameters
  • max_threads (int) – Number of threads. Default is 2.

  • thread_name_prefix (string) – Thread name prefix

  • storage_access (list of Staging) – Specifications for accessing data this executor remotely.

  • managed (bool) – If True, parsl will control dynamic scaling of this executor, and be responsible. Otherwise, this is managed by the user.

__init__(label: str = 'threads', max_threads: int = 2, thread_name_prefix: str = '', storage_access: List[Any] = None, working_dir: Optional[str] = None, managed: bool = True)[source]

Methods

__init__([label, max_threads, ...])

create_monitoring_info(status)

Create a monitoring message for each block based on the poll status.

handle_errors(error_handler, status)

This method is called by the error management infrastructure after a status poll.

monitor_resources()

Resource monitoring sometimes deadlocks when using threads, so this function returns false to disable it.

scale_in(blocks)

Scale in the number of active blocks by specified amount.

scale_out([workers])

Scales out the number of active workers by 1.

set_bad_state_and_fail_all(exception)

Allows external error handlers to mark this executor as irrecoverably bad and cause all tasks submitted to it now and in the future to fail.

shutdown([block])

Shutdown the ThreadPool.

start()

Start the executor.

status()

Return the status of all jobs/blocks currently known to this executor.

submit(func, resource_specification, *args, ...)

Submits work to the thread pool.

Attributes

bad_state_is_set

Returns true if this executor is in an irrecoverable error state.

error_management_enabled

Indicates whether worker error management is supported by this executor.

executor_exception

Returns an exception that indicates why this executor is in an irrecoverable state.

hub_address

Address to the Hub for monitoring.

hub_port

Port to the Hub for monitoring.

label

provider

radio_mode

run_dir

Path to the run directory.

scaling_enabled

Specify if scaling is enabled.

status_polling_interval

Returns the interval, in seconds, at which the status method should be called.

tasks

Contains a dictionary mapping task IDs to the corresponding Future objects for all tasks that have been submitted to this executor.

monitor_resources()[source]

Resource monitoring sometimes deadlocks when using threads, so this function returns false to disable it.

scale_in(blocks)[source]

Scale in the number of active blocks by specified amount.

This method is not implemented for threads and will raise the error if called.

Raises

NotImplemented exception

scale_out(workers=1)[source]

Scales out the number of active workers by 1.

This method is notImplemented for threads and will raise the error if called.

Raises

NotImplemented exception

property scaling_enabled[source]

Specify if scaling is enabled.

The callers of ParslExecutors need to differentiate between Executors and Executors wrapped in a resource provider

shutdown(block=True)[source]

Shutdown the ThreadPool. The underlying concurrent.futures thread pool implementation will not terminate tasks that are being executed, because it does not provide a mechanism to do that. With block set to false, this will return immediately and it will appear as if the DFK is shut down, but the python process will not be able to exit until the thread pool has emptied out by task completions. In either case, this can be a very long wait.

Kwargs:
  • block (Bool): To block for confirmations or not

start()[source]

Start the executor.

Any spin-up operations (for example: starting thread pools) should be performed here.

submit(func, resource_specification, *args, **kwargs)[source]

Submits work to the thread pool.

This method is simply pass through and behaves like a submit call as described here Python docs: