parsl.data_provider.data_manager.DataManager

class parsl.data_provider.data_manager.DataManager(dfk: DataFlowKernel)

The DataManager is responsible for transferring input and output data.

__init__(dfk: DataFlowKernel) → None

Initialize the DataManager.

Parameters

dfk (-) – The DataFlowKernel that this DataManager is managing data for.

Methods

__init__(dfk)	Initialize the DataManager.
optionally_stage_in(input, func, executor)
replace_task(file, func, executor)	This will give staging providers the chance to wrap (or replace entirely!) the task function.
replace_task_stage_out(file, func, executor)	This will give staging providers the chance to wrap (or replace entirely!) the task function.
stage_in(file, input, executor)	Transport the input from the input source to the executor, if it is file-like, returning a DataFuture that wraps the stage-in operation.
stage_out(file, executor, app_fu)	Transport the file from the local filesystem to the remote Globus endpoint.

optionally_stage_in(input, func, executor)

replace_task(file: parsl.data_provider.files.File, func: Callable, executor: str) → Callable

This will give staging providers the chance to wrap (or replace entirely!) the task function.

replace_task_stage_out(file: parsl.data_provider.files.File, func: Callable, executor: str) → Callable

This will give staging providers the chance to wrap (or replace entirely!) the task function.

stage_in(file: parsl.data_provider.files.File, input: Any, executor: str) → Any

Transport the input from the input source to the executor, if it is file-like, returning a DataFuture that wraps the stage-in operation.

If no staging in is required - because the file parameter is not file-like, then return that parameter unaltered.

Parameters

• self (-) –

• input (-) – input to stage in. If this is a File or a DataFuture, stage in tasks will be launched with appropriate dependencies. Otherwise, no stage-in will be performed.

• executor (-) – an executor the file is going to be staged in to.

stage_out(file: parsl.data_provider.files.File, executor: str, app_fu: concurrent.futures._base.Future) → Optional[concurrent.futures._base.Future]

Transport the file from the local filesystem to the remote Globus endpoint.

This function returns either a Future which should complete when the stageout is complete, or None, if no staging needs to be waited for.

Parameters

• self (-) –

• file (-) –

• executor (-) –

• app_fu (-) – complete before stageout begins.