@parallel¶

@parallel ( [ [job_params, ...], [job_params, ...]...] | parameter_generating_function)¶

Purpose:

To apply the (task) function to a set of parameters in parallel without file dependency checking.

Most useful allied to @check_if_uptodate()

Example:

from ruffus import *
parameters = [
                 ['A', 1, 2], # 1st job
                 ['B', 3, 4], # 2nd job
                 ['C', 5, 6], # 3rd job
             ]
@parallel(parameters)
def parallel_task(name, param1, param2):
    sys.stderr.write("    Parallel task %s: " % name)
    sys.stderr.write("%d + %d = %d\\n" % (param1, param2, param1 + param2))

pipeline_run([parallel_task])

Parameters:

job_params:
Requires a sequence of parameters, one set for each job.

Each set of parameters can be one or more items in a sequence which will be passed to the decorated task function iteratively (or in parallel)

For example:
parameters = [
                 ['A', 1, 2], # 1st job
                 ['B', 3, 4], # 2nd job
                 ['C', 5, 6], # 3rd job
             ]
@parallel(parameters)
def parallel_task(name, param1, param2):
    pass
Will result in the following function calls:
parallel_task('A', 1, 2)
parallel_task('B', 3, 4)
parallel_task('C', 5, 6)

parameter_generating_function

A generator yielding set of parameters (as above) in turn and on the fly

A function returning a sequence of parameter sets, as above