parsl.dataflow.dflow.DataFlowKernel

class parsl.dataflow.dflow.DataFlowKernel(config=Config(app_cache=True, checkpoint_files=None, checkpoint_mode=None, checkpoint_period=None, executors=[ThreadPoolExecutor(label='threads', managed=True, max_threads=2, storage_access=None, thread_name_prefix='', working_dir=None)], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=0, run_dir='runinfo', strategy='simple', usage_tracking=False))

The DataFlowKernel adds dependency awareness to an existing executor.

It is responsible for managing futures, such that when dependencies are resolved, pending tasks move to the runnable state.

Here is a simplified diagram of what happens internally:

 User             |        DFK         |    Executor
----------------------------------------------------------
                  |                    |
       Task-------+> +Submit           |
     App_Fu<------+--|                 |
                  |  Dependencies met  |
                  |         task-------+--> +Submit
                  |        Ex_Fu<------+----|

__init__(config=Config(app_cache=True, checkpoint_files=None, checkpoint_mode=None, checkpoint_period=None, executors=[ThreadPoolExecutor(label='threads', managed=True, max_threads=2, storage_access=None, thread_name_prefix='', working_dir=None)], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=0, run_dir='runinfo', strategy='simple', usage_tracking=False))

Initialize the DataFlowKernel.

Parameters

config (Config) – A specification of all configuration options. For more details see the :class:~`parsl.config.Config` documentation.

Methods

__init__([config])	Initialize the DataFlowKernel.
add_executors(executors)
atexit_cleanup()
check_staging_inhibited(kwargs)
checkpoint([tasks])	Checkpoint the dfk incrementally to a checkpoint file.
cleanup()	DataFlowKernel cleanup.
handle_app_update(task_id, future)	This function is called as a callback when an AppFuture is in its final state.
handle_exec_update(task_id, future)	This function is called only as a callback from an execution attempt reaching a final state (either successfully or failing).
handle_join_update(outer_task_id, …)
launch_if_ready(task_id)	launch_if_ready will launch the specified task, if it is ready to run (for example, without dependencies, and in pending state).
launch_task(task_id, executable, *args, **kwargs)	Handle the actual submission of the task to the executor layer.
load_checkpoints(checkpointDirs)	Load checkpoints from the checkpoint files into a dictionary.
log_task_states()
sanitize_and_wrap(task_id, args, kwargs)	This function should be called only when all the futures we track have been resolved.
submit(func, app_args[, executors, cache, …])	Add task to the dataflow system.
wait_for_current_tasks()	Waits for all tasks in the task list to be completed, by waiting for their AppFuture to be completed.
wipe_task(task_id)	Remove task with task_id from the internal tasks table

Attributes

config

Returns the fully initialized config that the DFK is actively using.