parsl.data_provider.data_manager.DataManager

class parsl.data_provider.data_manager.DataManager(dfk: DataFlowKernel)[source]

The DataManager is responsible for transferring input and output data.

__init__(dfk: DataFlowKernel) → None[source]

Initialize the DataManager.

Parameters:

dfk (-) – The DataFlowKernel that this DataManager is managing data for.

Methods

__init__(dfk)

Initialize the DataManager.

optionally_stage_in(input, func, executor)

replace_task(file, func, executor)

This will give staging providers the chance to wrap (or replace entirely!) the task function.

replace_task_stage_out(file, func, executor)

This will give staging providers the chance to wrap (or replace entirely!) the task function.

stage_in(file, input, executor)

Transport the input from the input source to the executor, if it is file-like, returning a DataFuture that wraps the stage-in operation.

stage_out(file, executor, app_fu)

Transport the file from the local filesystem to the remote Globus endpoint.
optionally_stage_in(input, func, executor)[source]
replace_task(file: File, func: Callable, executor: str) → Callable[source]

This will give staging providers the chance to wrap (or replace entirely!) the task function.

replace_task_stage_out(file: File, func: Callable, executor: str) → Callable[source]

This will give staging providers the chance to wrap (or replace entirely!) the task function.

stage_in(file: File, input: Any, executor: str) → Any[source]

Transport the input from the input source to the executor, if it is file-like, returning a DataFuture that wraps the stage-in operation.

If no staging in is required - because the file parameter is not file-like, then return that parameter unaltered.

Parameters:

  • self (-)

  • input (-) – input to stage in. If this is a File or a DataFuture, stage in tasks will be launched with appropriate dependencies. Otherwise, no stage-in will be performed.

  • executor (-) – an executor the file is going to be staged in to.

stage_out(file: File, executor: str, app_fu: Future) → Future | None[source]

Transport the file from the local filesystem to the remote Globus endpoint.

This function returns either a Future which should complete when the stageout is complete, or None, if no staging needs to be waited for.

Parameters:

  • self (-)

  • file (-)

  • executor (-)

  • app_fu (-) – complete before stageout begins.