parsl.data_provider.staging.Staging

class parsl.data_provider.staging.Staging[source]

This class defines the interface for file staging providers.

For each file to be staged in, the data manager will present the file to each configured Staging provider in turn: first, it will ask if the provider can stage this file by calling can_stage_in, and if so, it will call both stage_in and replace_task to give the provider the opportunity to perform staging.

For each file to be staged out, the data manager will follow the same pattern using the corresponding stage out methods of this class.

The default implementation of this class rejects all files, and performs no staging actions.

To implement a concrete provider, one or both of the can_stage_* methods should be overridden to match the appropriate files, and then the corresponding stage_* and/or replace_task* methods should be implemented.

can_stage_in(file: parsl.data_provider.files.File) → bool[source]

Given a File object, decide if this staging provider can stage the file. Usually this is be based on the URL scheme, but does not have to be. If this returns True, then other methods of this Staging object will be called to perform the staging.

can_stage_out(file: parsl.data_provider.files.File) → bool[source]

Like can_stage_in, but for staging out.

replace_task(dm: DataManager, executor: str, file: parsl.data_provider.files.File, func: Callable) → Optional[Callable][source]

For a file to be staged in, either return a replacement app function, which usually should be the original app function wrapped in staging code.

replace_task_stage_out(dm: DataManager, executor: str, file: parsl.data_provider.files.File, func: Callable) → Optional[Callable][source]
stage_in(dm: DataManager, executor: str, file: parsl.data_provider.files.File, parent_fut: Optional[concurrent.futures._base.Future]) → Optional[parsl.app.futures.DataFuture][source]

For a given file, either return a DataFuture to substitute for this file, or return None to perform no substitution

stage_out(dm: DataManager, executor: str, file: parsl.data_provider.files.File, app_fu) → Optional[parsl.app.futures.DataFuture][source]