Structuring Parsl programs

While convenient to build simple Parsl programs as a single Python file, splitting a Parsl programs into multiple files and a Python module has significant benefits, including:

1. Better readability

2. Logical separation of components (e.g., apps, config, and control logic)

3. Ease of reuse of components

Large applications that use Parsl often divide into several core components:

The following sections use an example where each component is in a separate file:

examples/logic.py
examples/app.py
examples/config.py
examples/__init__.py
run.py
pyproject.toml

Run the application by first installing the Python library and then executing the “run.py” script.

pip install .  # Install module so it can be imported by workers
python run.py

Core application logic

The core application logic should be developed without any deference to Parsl. Implement capabilities, write unit tests, and prepare documentation in which ever way works best for the problem at hand.

Parallelization with Parsl will be easy if the software already follows best practices.

The example defines a function to convert a single integer into binary.

library/logic.py

from typing import Tuple


def convert_to_binary(x: int) -> Tuple[bool, ...]:
    """Convert a nonnegative integer into a binary

    Args:
        x: Number to be converted
    Returns:
        The binary number represented as list of booleans
    """
    if x < 0:
        raise ValueError('`x` must be nonnegative')
    bin_as_string = bin(x)
    return tuple(i == '1' for i in bin_as_string[2:])

Workflow functions

Tasks within a workflow may require unique combinations of core functions. Functions to be run in parallel must also meet specific requirements that may complicate writing the core logic effectively. As such, separating functions to be used as Apps is often beneficial.

The example includes a function to convert many integers into binary.

Key points to note:

• It is not necessary to have import statements inside the function. Parsl will serialize this function by reference, as described in Functions from Modules.

• The function is not yet marked as a Parsl PythonApp. Keeping Parsl out of the function definitions simplifies testing because you will not need to run Parsl when testing the code.

• Advanced: Consider including Parsl decorators in the library if using complex workflow patterns, such as join apps or functions which take special arguments.

library/app.py

"""Functions used as part of the workflow"""
from typing import List, Tuple

from .logic import convert_to_binary


def convert_many_to_binary(xs: List[int]) -> List[Tuple[bool, ...]]:
    """Convert a list of nonnegative integers to binary"""
    return [convert_to_binary(x) for x in xs]

Parsl configuration functions

Create Parsl configurations specific to your application needs as functions. While not necessary, including the Parsl configuration functions inside the module ensures they can be imported into other scripts easily.

Generating Parsl Config objects from a function makes it possible to change the configuration without editing the module.

The example function provides a configuration suited for a single node.

library/config.py

from parsl.config import Config
from parsl.executors import HighThroughputExecutor
from parsl.providers import LocalProvider


def make_local_config(cores_per_worker: int = 1) -> Config:
    """Generate a configuration which runs all tasks on the local system

    Args:
        cores_per_worker: Number of cores to dedicate for each task
    Returns:
        Configuration object with the requested settings
    """
    return Config(
        executors=[
            HighThroughputExecutor(
                label="htex_local",
                cores_per_worker=cores_per_worker,
                cpu_affinity='block',
                provider=LocalProvider(),
            )
        ],
    )

Orchestration Scripts

The last file defines the workflow itself.

Such orchestration scripts, at minimum, perform at least four tasks:

1. Load execution options using a tool like argparse.

2. Prepare workflow functions for execution by creating PythonApp wrappers over each function.

3. Create configuration then start Parsl with the parsl.load() function.

4. Launch tasks and retrieve results depending on the needs of the application.

An example run script is as follows

run.py

from argparse import ArgumentParser

import parsl

from library.config import make_local_config
from library.app import convert_many_to_binary
from parsl.app.python import PythonApp

# Protect the script from running twice.
#  See "Safe importing of main module" in Python multiprocessing docs
#  https://docs.python.org/3/library/multiprocessing.html#multiprocessing-programming
if __name__ == "__main__":
    # Get user instructions
    parser = ArgumentParser()
    parser.add_argument('--numbers-per-batch', default=8, type=int)
    parser.add_argument('numbers', nargs='+', type=int)
    args = parser.parse_args()

    # Prepare the workflow functions
    convert_app = PythonApp(convert_many_to_binary, cache=False)

    # Load the configuration
    #  As a context manager so resources are shutdown on exit
    with parsl.load(make_local_config()):

        # Spawn tasks
        futures = [
            convert_app(args.numbers[start:start + args.numbers_per_batch])
            for start in range(0, len(args.numbers), args.numbers_per_batch)
        ]

        # Retrieve task results
        for future in futures:
            for x, b in zip(future.task_record['args'][0], future.result()):
                print(f'{x} -> {"".join("1" if i else "0" for i in b)}')