Parsl - Parallel Scripting Library¶
Parsl extends parallelism in Python beyond a single computer.
You can use Parsl just like Python’s parallel executors but across multiple cores and nodes. However, the real power of Parsl is in expressing multi-step workflows of functions. Parsl lets you chain functions together and will launch each function as inputs and computing resources are available.
import parsl from parsl import python_app # Start Parsl on a single computer parsl.load() # Make functions parallel by decorating them @python_app def f(x): return x + 1 @python_app def g(x): return x * 2 # These functions now return Futures, and can be chained future = f(1) assert future.result() == 2 future = g(f(1)) assert future.result() == 4
Start with the configuration quickstart to learn how to tell Parsl how to use your computing resource, then explore the parallel computing patterns to determine how to use parallelism best in your application.
Parsl is an open-source code, and available on GitHub: https://github.com/parsl/parsl/
Why use Parsl?¶
Parsl is Python¶
Everything about a Parsl program is written in Python. Parsl follows Python’s native parallelization approach and functions, how they combine into workflows, and where they run are all described in Python.
Parsl works everywhere¶
Parsl can run parallel functions on a laptop and the world’s fastest supercomputers. Scaling from laptop to supercomputer is often as simple as changing the resource configuration.
Parsl is flexible¶
Parsl supports many kinds of applications. Functions can be pure Python or invoke external codes, be single- or multi-threaded or GPUs.
Parsl handles data¶
Parsl has first-class support for workflows involving files. Data will be automatically moved between workers, even if they reside on different filesystems.
Parsl is fast¶
Parsl was built for speed. Workflows can manage tens of thousands of parallel tasks and process thousands of tasks per second.
Parsl is a community¶
Parsl is part of a large, experienced community.
The Parsl Project was launched by researchers with decades of experience in workflows as part of a National Science Foundation project to create sustainable research software.
The Parsl team is guided by the community through its GitHub, conversations on Slack, Bi-Weekly developer calls, and engagement with the Workflows Community Initiative.
Table of Contents¶
- Parsl tutorial
- User guide
- How can I debug a Parsl script?
- How can I view outputs and errors from apps?
- How can I make an App dependent on multiple inputs?
- Can I pass any Python object between apps?
- How do I specify where apps should be run?
- Workers do not connect back to Parsl
- Remote execution fails with SystemError(unknown opcode)
- Parsl complains about missing packages
- zmq.error.ZMQError: Invalid argument
- How do I run code that uses Python2.X?
- Parsl hangs
- How can I start a Jupyter notebook over SSH?
- How can I sync my conda environment and Jupyter environment?
- Addressing SerializationError
- How do I cite Parsl?
- API Reference guide
- Developer documentation