Indicator Objects¶

How ruffus disambiguates certain parameters to decorators.

They are like keyword arguments in python, a little more verbose but they make the syntax much simpler.

Indicator objects are also “self-documenting” so you can see exactly what is happening clearly.

formatter¶

formatter([ regex | None , regex | None...])

The optional enclosed parameters are a python regular expression strings
Each regular expression matches a corresponding Input file name string
formatter parses each file name string into path and regular expression components
Parsing fails altogether if the regular expression is not matched

Path components include:

basename: The base name excluding extension, "file.name"

ext : The extension, ".ext"

path : The dirname, "/directory/to/a"

subdir : A list of sub-directories in the path in reverse order, ["a", "to", "directory", "/"]

subpath : A list of descending sub-paths in reverse order, ["/directory/to/a", "/directory/to", "/directory", "/"]

The replacement string refers to these components using python string.format style curly braces. {NAME}

We refer to an element from the Nth input string by index, for example:

"{ext[0]}" is the extension of the first input string.

"{basename[1]}" is the basename of the second input string.

"{basename[1][0:3]}" are the first three letters from the basename of the second input string.

Used by:

@transform example:

from ruffus import *

#   create initial file pairs
@originate([   ['job1.a.start', 'job1.b.start'],
               ['job2.a.start', 'job2.b.start'],
               ['job3.a.start', 'job3.c.start']    ])
def create_initial_file_pairs(output_files):
    for output_file in output_files:
        with open(output_file, "w") as oo: pass


#---------------------------------------------------------------
#
#   formatter
#
@transform(create_initial_file_pairs,                               # Input

            formatter(".+/job(?P<JOBNUMBER>\d+).a.start",           # Extract job number
                      ".+/job[123].b.start"),                       # Match only "b" files

            ["{path[0]}/jobs{JOBNUMBER[0]}.output.a.1",             # Replacement list
             "{path[1]}/jobs{JOBNUMBER[0]}.output.b.1"])
def first_task(input_files, output_parameters):
    print "input_parameters = ", input_files
    print "output_parameters = ", output_parameters


#
#       Run
#
pipeline_run(verbose=0)

This produces:

input_parameters  =  ['job1.a.start',
                      'job1.b.start']
output_parameters =  ['/home/lg/src/temp/jobs1.output.a.1',
                      '/home/lg/src/temp/jobs1.output.b.1', 45]

input_parameters  =  ['job2.a.start',
                      'job2.b.start']
output_parameters =  ['/home/lg/src/temp/jobs2.output.a.1',
                      '/home/lg/src/temp/jobs2.output.b.1', 45]

@permutations example:

Combinatoric decorators such as @product or @product behave much like nested for loops in enumerating, combining, and permutating the original sets of inputs.

The replacement strings require an extra level of indirection to refer to parsed components:

from ruffus import *
from ruffus.combinatorics import *

#   create initial files
@originate([ 'a.start', 'b.start', 'c.start'])
def create_initial_files(output_file):
    with open(output_file, "w") as oo: pass


#---------------------------------------------------------------
#
#   formatter
#
@permutations(create_initial_files,                                           # Input

            formatter("(.start)$"),                                           # match input file in permutations
            2,

            "{path[0][0]}/{basename[0][0]}_vs_{basename[1][0]}.product",    # Output Replacement string
            "{path[0][0]}",                                                 # path for 1st set of files, 1st file name
            ["{basename[0][0]}",                                            # basename for 1st set of files, 1st file name
             "{basename[1][0]}"])                                           # basename for 2nd set of files, 1st file name
def product_task(input_file, output_parameter, shared_path, basenames):
    print "input_parameter  = ", input_file
    print "output_parameter = ", output_parameter
    print "shared_path      = ", shared_path
    print "basenames        = ", basenames


#
#       Run
#
pipeline_run(verbose=0)

This produces:

>>> pipeline_run(verbose=0)
input_parameter  =  ('a.start', 'b.start')
output_parameter =  /home/lg/src/oss/ruffus/a_vs_b.product
shared_path      =  /home/lg/src/oss/ruffus
basenames        =  ['a', 'b']

input_parameter  =  ('a.start', 'c.start')
output_parameter =  /home/lg/src/oss/ruffus/a_vs_c.product
shared_path      =  /home/lg/src/oss/ruffus
basenames        =  ['a', 'c']

input_parameter  =  ('b.start', 'a.start')
output_parameter =  /home/lg/src/oss/ruffus/b_vs_a.product
shared_path      =  /home/lg/src/oss/ruffus
basenames        =  ['b', 'a']

input_parameter  =  ('b.start', 'c.start')
output_parameter =  /home/lg/src/oss/ruffus/b_vs_c.product
shared_path      =  /home/lg/src/oss/ruffus
basenames        =  ['b', 'c']

input_parameter  =  ('c.start', 'a.start')
output_parameter =  /home/lg/src/oss/ruffus/c_vs_a.product
shared_path      =  /home/lg/src/oss/ruffus
basenames        =  ['c', 'a']

input_parameter  =  ('c.start', 'b.start')
output_parameter =  /home/lg/src/oss/ruffus/c_vs_b.product
shared_path      =  /home/lg/src/oss/ruffus
basenames        =  ['c', 'b']

suffix¶

suffix( string )

The enclosed parameter is a string which must match exactly to the end of a file name.
Used by:

@transform

Example:
#
#   Transforms ``*.c`` to ``*.o``::
#
@transform(previous_task, suffix(".c"), ".o")
def compile(infile, outfile):
    pass

regex¶

regex( regular_expression )

The enclosed parameter is a python regular expression string, which must be wrapped in a regex indicator object.

See python regular expression (re) documentation for details of regular expression syntax

Used by:

@transform

@subdivide

@collate

The deprecated @files_re
Example:
@transform(previous_task, regex(r".c$"), ".o")
def compile(infile, outfile):
    pass

add_inputs¶

add_inputs( input_file_pattern )

The enclosed parameter(s) are pattern strings or a nested structure which is added to the input for each job.

Used by:

@transform

@collate

@subdivide

Example @transform with suffix(...)
A common task in compiling C code is to include the corresponding header file for the source. To compile *.c to *.o, adding *.h and the common header universal.h:
@transform(["1.c", "2.c"], suffix(".c"), add_inputs([r"\1.h", "universal.h"]),  ".o")
def compile(infile, outfile):
    # do something here
    pass
The starting files names are 1.c and 2.c.

suffix(".c") matches ”.c” so \1 stands for the unmatched prefices "1" and "2"
This will result in the following functional calls:
compile(["1.c", "1.h", "universal.h"], "1.o")
compile(["2.c", "2.h", "universal.h"], "2.o")
A string like universal.h in add_inputs will added as is. r"\1.h", however, performs suffix substitution, with the special form r"\1" matching everything up to the suffix. Remember to ‘escape’ r"\1" otherwise Ruffus will complain and throw an Exception to remind you. The most convenient way is to use a python “raw” string.
Example of add_inputs(...) with regex(...)
The suffix match (suffix(...)) is exactly equivalent to the following code using regular expression (regex(...)):
@transform(["1.c", "2.c"], regex(r"^(.+)\.c$"), add_inputs([r"\1.h", "universal.h"]),  r"\1.o")
def compile(infile, outfile):
    # do something here
    pass
The suffix(..) code is much simpler but the regular expression allows more complex substitutions.
add_inputs(...) preserves original inputs
add_inputs nests the the original input parameters in a list before adding additional dependencies.
This can be seen in the following example:
@transform([    ["1.c", "A.c", 2]
                ["2.c", "B.c", "C.c", 3]],
                suffix(".c"), add_inputs([r"\1.h", "universal.h"]),  ".o")
def compile(infile, outfile):
    # do something here
    pass
This will result in the following functional calls:
compile([["1.c", "A.c", 2],        "1.h", "universal.h"], "1.o")
compile([["3.c", "B.c", "C.c", 3], "2.h", "universal.h"], "2.o")
The original parameters are retained unchanged as the first item in a list

inputs¶

inputs( input_file_pattern )

Used by:

@transform

@collate

@subdivide

The enclosed single parameter is a pattern string or a nested structure which is used to construct the input for each job.

If more than one argument is supplied to inputs, an exception will be raised.

Use a tuple or list (as in the following example) to send multiple input arguments to each job.

Used by:

The advanced form of @transform

inputs(...) replaces original inputs
inputs(...) allows the original input parameters to be replaced wholescale.
This can be seen in the following example:
@transform([    ["1.c", "A.c", 2]
                ["2.c", "B.c", "C.c", 3]],
                suffix(".c"), inputs([r"\1.py", "docs.rst"]),  ".pyc")
def compile(infile, outfile):
    # do something here
    pass
This will result in the following functional calls:
compile(["1.py", "docs.rst"], "1.pyc")
compile(["2.py", "docs.rst"], "2.pyc")
In this example, the corresponding python files have been sneakily substituted without trace in the place of the C source files.

mkdir¶

mkdir( directory_name1 , [ directory_name2 , ...] )

The enclosed parameter is a directory name or a sequence of directory names. These directories will be created as part of the prerequisites of running a task.
Used by:

@follows

Example:
@follows(mkdir("/output/directory"))
def task():
    pass

touch_file¶

touch_file( file_name )

The enclosed parameter is a file name. This file will be touch-ed after a task is executed.

This will change the date/time stamp of the file_name to the current date/time. If the file does not exist, an empty file will be created.
Used by:

@posttask

Example:
@posttask(touch_file("task_completed.flag"))
@files(None, "a.1")
def do_task(input_file, output_file):
    pass

output_from¶

output_from ( file_name_string1 [, file_name_string1 , ...] )

Indicates that any enclosed strings are not file names but refer to task functions.
Used by:

@split

@transform

@merge

@collate

@subdivide

@product

@permutations

@combinations

@combinations_with_replacement

@files

Example:
@split(["a.file", ("b.file", output_from("task1", 76, "task2"))], "*.split")
def task2(input, output):
    pass
is equivalent to:
@split(["a.file", ("b.file", (task1, 76, task2))], "*.split")
def task2(input, output):
    pass

combine¶

combine( arguments )

Warning

This is deprecated syntax.

Please do not use!

@merge and @collate are more powerful and have straightforward syntax.

Indicates that the inputs of @files_re will be collated or summarised into outputs by category. See the Manual or :ref:` @collate <new_manual.collate>` for examples.
Used by:

@files_re

Example:
@files_re('*.animals',                           # inputs = all *.animal files
            r'mammals.([^.]+)',                  # regular expression
            combine(r'\1/animals.in_my_zoo'),    # single output file per species
            r'\1' )                              # species name
def capture_mammals(infiles, outfile, species):
    # summarise all animals of this species
    ""