API Reference guide

Core

parsl.app.app.python_app

Decorator function for making python apps.

parsl.app.app.bash_app

Decorator function for making bash apps.

parsl.app.app.join_app

Decorator function for making join apps

parsl.dataflow.futures.AppFuture

An AppFuture wraps a sequence of Futures which may fail and be retried.

parsl.dataflow.dflow.DataFlowKernelLoader

Manage which DataFlowKernel is active.

parsl.monitoring.MonitoringHub

Configuration

parsl.config.Config

Specification of Parsl configuration options.

parsl.set_stream_logger

Add a stream log handler.

parsl.set_file_logger

Add a file log handler.

parsl.addresses.address_by_hostname

Returns the hostname of the local host.

parsl.addresses.address_by_interface

Returns the IP address of the given interface name, e.g.

parsl.addresses.address_by_query

Finds an address for the local host by querying ipify.

parsl.addresses.address_by_route

Finds an address for the local host by querying the local routing table for the route to Google DNS.

parsl.utils.get_all_checkpoints

Finds the checkpoints from all runs in the rundir.

parsl.utils.get_last_checkpoint

Finds the checkpoint from the last run, if one exists.

Channels

parsl.channels.base.Channel

Channels are abstractions that enable ExecutionProviders to talk to resource managers of remote compute facilities.

parsl.channels.LocalChannel

This is not even really a channel, since opening a local shell is not heavy and done so infrequently that they do not need a persistent channel

parsl.channels.SSHChannel

SSH persistent channel.

parsl.channels.OAuthSSHChannel

SSH persistent channel.

parsl.channels.SSHInteractiveLoginChannel

SSH persistent channel.

Data management

parsl.app.futures.DataFuture

A datafuture points at an AppFuture.

parsl.data_provider.data_manager.DataManager

The DataManager is responsible for transferring input and output data.

parsl.data_provider.staging.Staging

This class defines the interface for file staging providers.

parsl.data_provider.files.File

The Parsl File Class.

parsl.data_provider.ftp.FTPSeparateTaskStaging

Performs FTP staging as a separate parsl level task.

parsl.data_provider.ftp.FTPInTaskStaging

Performs FTP staging as a wrapper around the application task.

parsl.data_provider.file_noop.NoOpFileStaging

parsl.data_provider.globus.GlobusStaging

Specification for accessing data on a remote executor via Globus.

parsl.data_provider.http.HTTPSeparateTaskStaging

A staging provider that Performs HTTP and HTTPS staging as a separate parsl-level task.

parsl.data_provider.http.HTTPInTaskStaging

A staging provider that performs HTTP and HTTPS staging as in a wrapper around each task.

parsl.data_provider.rsync.RSyncStaging

This staging provider will execute rsync on worker nodes to stage in files from a remote location.

Executors

parsl.executors.base.ParslExecutor

Executors are abstractions that represent available compute resources to which you could submit arbitrary App tasks.

parsl.executors.status_handling.BlockProviderExecutor

A base class for executors which scale using blocks.

parsl.executors.ThreadPoolExecutor

A thread-based executor.

parsl.executors.HighThroughputExecutor

Executor designed for cluster-scale

parsl.executors.WorkQueueExecutor

Executor to use Work Queue batch system

parsl.executors.taskvine.TaskVineExecutor

Executor to use TaskVine dynamic workflow system

parsl.executors.FluxExecutor

Executor that uses Flux to schedule and run jobs.

parsl.executors.radical.RadicalPilotExecutor

Executor is designed for executing heterogeneous tasks

Launchers

parsl.launchers.base.Launcher

Launchers are basically wrappers for user submitted scripts as they are submitted to a specific execution resource.

parsl.launchers.SimpleLauncher

Does no wrapping.

parsl.launchers.SingleNodeLauncher

Worker launcher that wraps the user's command with the framework to launch multiple command invocations in parallel.

parsl.launchers.SrunLauncher

Worker launcher that wraps the user's command with the SRUN launch framework to launch multiple cmd invocations in parallel on a single job allocation.

parsl.launchers.AprunLauncher

Worker launcher that wraps the user's command with the Aprun launch framework to launch multiple cmd invocations in parallel on a single job allocation

parsl.launchers.SrunMPILauncher

Launches as many workers as MPI tasks to be executed concurrently within a block.

parsl.launchers.GnuParallelLauncher

Worker launcher that wraps the user's command with the framework to launch multiple command invocations via GNU parallel sshlogin.

parsl.launchers.MpiExecLauncher

Worker launcher that wraps the user's command with the framework to launch multiple command invocations via mpiexec.

parsl.launchers.JsrunLauncher

Worker launcher that wraps the user's command with the Jsrun launch framework to launch multiple cmd invocations in parallel on a single job allocation

parsl.launchers.WrappedLauncher

Wraps the command by prepending commands before a user's command

Providers

parsl.providers.AdHocProvider

Ad-hoc execution provider

parsl.providers.AWSProvider

A provider for using Amazon Elastic Compute Cloud (EC2) resources.

parsl.providers.CobaltProvider

Cobalt Execution Provider

parsl.providers.CondorProvider

HTCondor Execution Provider.

parsl.providers.GoogleCloudProvider

A provider for using resources from the Google Compute Engine.

parsl.providers.GridEngineProvider

A provider for the Grid Engine scheduler.

parsl.providers.LocalProvider

Local Execution Provider

parsl.providers.LSFProvider

LSF Execution Provider

parsl.providers.GridEngineProvider

A provider for the Grid Engine scheduler.

parsl.providers.SlurmProvider

Slurm Execution Provider

parsl.providers.TorqueProvider

Torque Execution Provider

parsl.providers.KubernetesProvider

Kubernetes execution provider

parsl.providers.PBSProProvider

PBS Pro Execution Provider

parsl.providers.base.ExecutionProvider

Execution providers are responsible for managing execution resources that have a Local Resource Manager (LRM).

parsl.providers.cluster_provider.ClusterProvider

This class defines behavior common to all cluster/supercompute-style scheduler systems.

Batch jobs

parsl.jobs.states.JobState

Defines a set of states that a job can be in

parsl.jobs.states.JobStatus

Encapsulates a job state together with other details:

parsl.jobs.error_handlers.noop_error_handler

parsl.jobs.error_handlers.simple_error_handler

parsl.jobs.error_handlers.windowed_error_handler

Exceptions

parsl.app.errors.AppBadFormatting

An error raised during formatting of a bash function.

parsl.app.errors.AppException

An error raised during execution of an app.

parsl.app.errors.AppTimeout

An error raised during execution of an app when it exceeds its allotted walltime.

parsl.app.errors.BadStdStreamFile

Error raised due to bad filepaths specified for STDOUT/ STDERR.

parsl.app.errors.BashAppNoReturn

Bash app returned no string.

parsl.app.errors.BashExitFailure

A non-zero exit code returned from a @bash_app

parsl.app.errors.MissingOutputs

Error raised at the end of app execution due to missing output files.

parsl.app.errors.ParslError

Base class for all exceptions.

parsl.errors.ConfigurationError

Raised when a component receives an invalid configuration.

parsl.errors.OptionalModuleMissing

Error raised when a required module is missing for a optional/extra component

parsl.executors.errors.ExecutorError

Base class for executor related exceptions.

parsl.executors.errors.ScalingFailed

Scaling failed due to error in Execution provider.

parsl.executors.errors.BadMessage

Mangled/Poorly formatted/Unsupported message received

parsl.dataflow.errors.DataFlowException

Base class for all exceptions.

parsl.dataflow.errors.BadCheckpoint

Error raised at the end of app execution due to missing output files.

parsl.dataflow.errors.DependencyError

Error raised if an app cannot run because there was an error

parsl.dataflow.errors.JoinError

Error raised if apps joining into a join_app raise exceptions.

parsl.launchers.errors.BadLauncher

Error raised when an object of inappropriate type is supplied as a Launcher

parsl.providers.errors.ExecutionProviderException

Base class for all exceptions Only to be invoked when only a more specific error is not available.

parsl.providers.errors.ScaleOutFailed

Scale out failed in the submit phase on the provider side

parsl.providers.errors.SchedulerMissingArgs

Error raised when the template used to compose the submit script to the local resource manager is missing required arguments

parsl.providers.errors.ScriptPathError

Error raised when the template used to compose the submit script to the local resource manager is missing required arguments

parsl.channels.errors.ChannelError

Base class for all exceptions

parsl.channels.errors.BadHostKeyException

SSH channel could not be created since server's host keys could not be verified

parsl.channels.errors.BadScriptPath

An error raised during execution of an app.

parsl.channels.errors.BadPermsScriptPath

User does not have permissions to access the script_dir on the remote site

parsl.channels.errors.FileExists

Push or pull of file over channel fails since a file of the name already exists on the destination.

parsl.channels.errors.AuthException

An error raised during execution of an app.

parsl.channels.errors.SSHException

if there was any other error connecting or establishing an SSH session

parsl.channels.errors.FileCopyException

File copy operation failed

parsl.executors.high_throughput.errors.WorkerLost

Exception raised when a worker is lost

parsl.executors.high_throughput.interchange.ManagerLost

Task lost due to manager loss.

parsl.serialize.errors.DeserializationError

Failure at the deserialization of results/exceptions from remote workers.

parsl.serialize.errors.SerializationError

Failure to serialize task objects.

Internal

parsl.app.app.AppBase

This is the base class that defines the two external facing functions that an App must define.

parsl.app.bash.BashApp

parsl.app.python.PythonApp

Extends AppBase to cover the Python App.

parsl.dataflow.dflow.DataFlowKernel

The DataFlowKernel adds dependency awareness to an existing executor.

parsl.dataflow.memoization.id_for_memo

This should return a byte sequence which identifies the supplied value for memoization purposes: for any two calls of id_for_memo, the byte sequence should be the same when the "same" value is supplied, and different otherwise.

parsl.dataflow.memoization.Memoizer

Memoizer is responsible for ensuring that identical work is not repeated.

parsl.dataflow.states.FINAL_STATES

States from which we will never move to another state, because the job has either definitively completed or failed.

parsl.dataflow.states.States

Enumerates the states a parsl task may be in.

parsl.dataflow.taskrecord.TaskRecord

This stores most information about a Parsl task

parsl.jobs.job_status_poller.JobStatusPoller

parsl.jobs.strategy.Strategy

Scaling strategy.

parsl.utils.Timer

This class will make a callback periodically, with a period specified by the interval parameter.