Parsl - Parallel Scripting Library
Parsl extends parallelism in Python beyond a single computer.
You can use Parsl just like Python’s parallel executors but across multiple cores and nodes. However, the real power of Parsl is in expressing multi-step workflows of functions. Parsl lets you chain functions together and will launch each function as inputs and computing resources are available.
from parsl import python_app
# Start Parsl on a single computer
# Make functions parallel by decorating them
return x + 1
return x * 2
# These functions now return Futures, and can be chained
future = f(1)
assert future.result() == 2
future = g(f(1))
assert future.result() == 4
Start with the configuration quickstart to learn how to tell Parsl how to use your computing resource, see if a template configuration for your supercomputer is already available, then explore the parallel computing patterns to determine how to use parallelism best in your application.
Parsl is an open-source code, and available on GitHub: https://github.com/parsl/parsl/
Why use Parsl?
Parsl is Python
Everything about a Parsl program is written in Python. Parsl follows Python’s native parallelization approach and functions, how they combine into workflows, and where they run are all described in Python.
Parsl works everywhere
Parsl can run parallel functions on a laptop and the world’s fastest supercomputers. Scaling from laptop to supercomputer is often as simple as changing the resource configuration. Parsl is tested on many of the top supercomputers.
Parsl is flexible
Parsl supports many kinds of applications. Functions can be pure Python or invoke external codes, be single- or multi-threaded or GPUs.
Parsl handles data
Parsl has first-class support for workflows involving files. Data will be automatically moved between workers, even if they reside on different filesystems.
Parsl is fast
Parsl was built for speed. Workflows can manage tens of thousands of parallel tasks and process thousands of tasks per second.
Parsl is a community
Parsl is part of a large, experienced community.
The Parsl Project was launched by researchers with decades of experience in workflows as part of a National Science Foundation project to create sustainable research software.
Table of Contents
- Parsl tutorial
- User guide
- Passing Python objects
- Staging data files
- MPI Apps
- Error handling
- Memoization and checkpointing
- Example parallel patterns
- Other useful patterns
- Structuring Parsl programs
- Lifted operators
- Join Apps
- Usage statistics collection
- Measuring performance with parsl-perf
- Historical: Performance and Scalability
- How can I debug a Parsl script?
- How can I view outputs and errors from apps?
- How can I make an App dependent on multiple inputs?
- Can I pass any Python object between apps?
- How do I specify where apps should be run?
- Workers do not connect back to Parsl
- Remote execution fails with SystemError(unknown opcode)
- Parsl complains about missing packages
- How do I run code that uses Python2.X?
- Parsl hangs
- How can I start a Jupyter notebook over SSH?
- How can I sync my conda environment and Jupyter environment?
- Addressing SerializationError
- How do I cite Parsl?
- API Reference guide
- Developer documentation