Workflows and Stages

class flowws.Workflow(stages, storage=None, scope={})

Specify a complete sequence of operations to perform.

Workflow objects specify a sequence of stages (operations to perform) and a storage object to use (which could be a database, archive file, or simply a directory on the filesystem). In addition to direct creation within python, Workflows can be deserialized from command line and JSON-based descriptions.

Stages are executed sequentially in the order they are given and each stage can pass information to later stages in a freeform way by settings elements of a scope, which is a dictionary of named values.

Parameters:

• stages – List of Stage objects specifying the operations to perform
• storage – Storage object specifying where results should be saved (default: create a DirectoryStorage using the current working directory)
• scope – Dictionary of key-value pairs specifying external input parameters

classmethod from_JSON(json_object, module_names='flowws_modules')

Construct a Workflow from a JSON object.

classmethod from_command(args=None, module_names='flowws_modules', scope={})

Construct a Workflow from a command-line description.

Stages are found based on setuptools entry_point specified under module_names.

Parameters:

• args – List of command-line arguments (list of strings)
• module_names – setuptools entry_point to use for module searches
• scope – Dictionary of initial key-value pairs to pass to child Stages

classmethod register_module(*args, module_names='flowws_modules', name=None)

Register a named module to be loaded inside from_JSON or other functions.

This method is intended to be used as a decorator for Stage classes in situations such as REPL loops or notebooks, where modules need to be deserialized without necessarily creating a standalone package and registering the endpoints through the setuptools machinery.

Examples:

@flowws.Workflow.register_module
class TestStage(flowws.Stage):
    pass

@flowws.Workflow.register_module(name='OverruledName')
class Stage(flowws.Stage):
    pass

run()

Run each stage inside this workflow.

Returns the scope after running all stages.

class flowws.Stage(**kwargs)

Base class for the building blocks of workflows.

Stage objects specify a discrete set of operations within a Workflow. Each Stage object has its own set of parameters and functionality that are then run in sequence when the workflow is run.

Stages can be instantiated within python by directly passing in arguments they take as keyword arguments, for example:

stages = [Initialize(seed=13), Run(parameter=1.5)]

Stages also can be instantiated from the command line using flowws.run (assuming they have been properly registered using setuptools entry_points):

python -m flowws.run Initialize --seed 13 Run --parameter 1.5

classmethod from_JSON(json_object)

Initialize this stage from a JSON representation

classmethod from_command(args)

Initialize this stage from a command-line description

run(scope, storage)

Run the contents of this stage

flowws.register_module(*args, module_names='flowws_modules', name=None)

Register a named module to be loaded inside from_JSON or other functions.

This method is intended to be used as a decorator for Stage classes in situations such as REPL loops or notebooks, where modules need to be deserialized without necessarily creating a standalone package and registering the endpoints through the setuptools machinery.

Examples:

@flowws.Workflow.register_module
class TestStage(flowws.Stage):
    pass

@flowws.Workflow.register_module(name='OverruledName')
class Stage(flowws.Stage):
    pass

flowws.try_to_import(pkg, name, current_pkg=None)

Import an attribute from a module, or return an error-producing fake.

This method is provided as a convenience for libraries that want to easily expose modules with a variety of prerequisite libraries without forcing the user to install prerequisites for the modules they do not use. The fake is produced if an import fails while importing the given package.

Parameters:

• name – Name of the attribute to return from the module
• pkg – Package name (can be relative)
• current_pkg – Name of the current package to use (i.e. if pkg is relative)

Returns:

Either the attribute from the successfully-imported module, or a fake module object that will produce an error if evaluated