parsl.executors.status_handling.BlockProviderExecutor¶
-
class
parsl.executors.status_handling.
BlockProviderExecutor
(provider: parsl.providers.provider_base.ExecutionProvider)[source]¶ A base class for executors which scale using blocks.
This base class is intended to help with executors which:
use blocks of workers to execute tasks
blocks of workers are launched on a batch system through an
ExecutionProvider
An implementing class should implement the abstract methods required by
ParslExecutor
to submit tasks, as well as BlockProviderExecutor abstract methods to provide the executor-specific command to start a block of workers (the_get_launch_command
method), and some basic scaling information (outstanding
andworkers_per_node
properties).This base class provides a
scale_out
method which will launch new blocks. It does not provide ascale_in
method, because scale-in behaviour is not well defined in the Parsl scaling model and so behaviour is left to individual executors.Parsl scaling will provide scaling between min_blocks and max_blocks by invoking scale_out, but it will not initialize the blocks requested by any init_blocks parameter. Subclasses must implement that behaviour themselves.
-
__init__
(provider: parsl.providers.provider_base.ExecutionProvider)[source]¶ Initialize self. See help(type(self)) for accurate signature.
Methods
__init__
(provider)Initialize self.
create_monitoring_info
(status)Create a monitoring message for each block based on the poll status.
handle_errors
(error_handler, status)This method is called by the error management infrastructure after a status poll.
monitor_resources
()Should resource monitoring happen for tasks on running on this executor?
scale_in
(blocks)Scale in method.
scale_out
([blocks])Scales out the number of blocks by “blocks”
set_bad_state_and_fail_all
(exception)Allows external error handlers to mark this executor as irrecoverably bad and cause all tasks submitted to it now and in the future to fail.
shutdown
()Shutdown the executor.
start
()Start the executor.
status
()Return status of all blocks.
submit
(func, resource_specification, *args, …)Submit.
Attributes
Returns true if this executor is in an irrecoverable error state.
Indicates whether worker error management is supported by this executor.
Returns an exception that indicates why this executor is in an irrecoverable state.
hub_address
Address to the Hub for monitoring.
hub_port
Port to the Hub for monitoring.
This should return the number of tasks that the executor has been given to run (waiting to run, and running now)
run_dir
Path to the run directory.
scaling_enabled
Specify if scaling is enabled.
Returns the interval, in seconds, at which the status method should be called.
Contains a dictionary mapping task IDs to the corresponding Future objects for all tasks that have been submitted to this executor.
-
property
bad_state_is_set
[source]¶ Returns true if this executor is in an irrecoverable error state. If this method returns true, :property:executor_exception should contain an exception indicating the cause.
-
property
error_management_enabled
[source]¶ Indicates whether worker error management is supported by this executor. Worker error management is done externally to the executor. However, the executor must implement certain status handling methods that allow this to function. These methods are:
:method:handle_errors :method:set_bad_state_and_fail_all
The basic idea of worker error management is that an external entity maintains a view of the state of the workers by calling :method:status() which is then processed to detect abnormal conditions. This can be done externally, as well as internally, through :method:handle_errors. If an entity external to the executor detects an abnormal condition, it can notify the executor using :method:set_bad_state_and_fail_all(exception).
Some of the scaffolding needed for implementing error management inside executors, including implementations for the status handling methods above, is available in :class:parsl.executors.status_handling.BlockProviderExecutor, which interested executors should inherit from. Noop versions of methods that are related to status handling and running parsl tasks through workers are implemented by :class:parsl.executors.status_handling.NoStatusHandlingExecutor.
-
property
executor_exception
[source]¶ Returns an exception that indicates why this executor is in an irrecoverable state.
-
handle_errors
(error_handler: parsl.dataflow.job_error_handler.JobErrorHandler, status: Dict[str, parsl.providers.provider_base.JobStatus]) → bool[source]¶ This method is called by the error management infrastructure after a status poll. The executor implementing this method is then responsible for detecting abnormal conditions based on the status of submitted jobs. If the executor does not implement any special error handling, this method should return False, in which case a generic error handling scheme will be used. :param error_handler: a reference to the generic error handler calling this method :param status: status of all jobs launched by this executor :return: True if this executor implements custom error handling, or False otherwise
-
abstract property
outstanding
[source]¶ This should return the number of tasks that the executor has been given to run (waiting to run, and running now)
-
set_bad_state_and_fail_all
(exception: Exception)[source]¶ Allows external error handlers to mark this executor as irrecoverably bad and cause all tasks submitted to it now and in the future to fail. The executor is responsible for checking :method:bad_state_is_set() in the :method:submit() method and raising the appropriate exception, which is available through :method:executor_exception().
-
property
status_polling_interval
[source]¶ Returns the interval, in seconds, at which the status method should be called. The assumption here is that, once initialized, an executor’s polling interval is fixed. In practice, at least given the current situation, the executor uses a single task provider and this method is a delegate to the corresponding method in the provider.
- Returns
the number of seconds to wait between calls to status() or zero if no polling should be done