See also

@transform( input, filter, output, [extras,...] )ΒΆ

Purpose:

Applies the task function to transform data from input to output files.

Output file names are specified from input, i.e. from the output of specified tasks, or a list of file names, or a glob matching pattern.

String replacement occurs either through suffix matches via suffix or the formatter or regex indicators.

Only out of date tasks (comparing input and output files) will be run

Simple Example

Transforms *.c to *.o:

@transform(input = ["1.c", "2.c"], filter = suffix(".c"), output = ".o")
def compile(infile, outfile):
    pass

Same example with a regular expression:

@transform(["1.c", "2.c"], regex(r".c$"), ".o")
def compile(infile, outfile):
    pass

Both result in the following function calls:

# 1.c -> 1.o
# 2.c -> 2.o
compile("1.c", "1.o")
compile("2.c", "2.o")

Escaping regular expression patterns

A string like universal.h in add_inputs will added as is. r"\1.h", however, performs suffix substitution, with the special form r"\1" matching everything up to the suffix. Remember to ‘escape’ r"\1" otherwise Ruffus will complain and throw an Exception to remind you. The most convenient way is to use a python “raw” string.

Parameters:

  • input = tasks_or_file_names

    can be a:

    1. Task / list of tasks (as in the example above).

      File names are taken from the output of the specified task(s)

    2. (Nested) list of file name strings.
      File names containing *[]? will be expanded as a glob.

      E.g.:"a.*" => "a.1", "a.2"

  • filter = suffix(suffix_string)

    must be wrapped in a suffix indicator object. The end of each input file name which matches suffix_string will be replaced by output.

    Input file names which do not match suffix_string will be ignored

    The non-suffix part of the match can be referred to using the "\1" pattern. This can be useful for putting the output in different directory, for example:

    @transform(["1.c", "2.c"], suffix(".c"), r"my_path/\1.o")
    def compile(infile, outfile):
        pass
    

    This results in the following function calls:

    # 1.c -> my_path/1.o
    # 2.c -> my_path/2.o
    compile("1.c", "my_path/1.o")
    compile("2.c", "my_path/2.o")
    

    For convenience and visual clarity, the "\1" can be omitted from the output parameter. However, the "\1" is mandatory for string substitutions in additional parameters,

    @transform(["1.c", "2.c"], suffix(".c"), [r"\1.o", ".o"], "Compiling \1", "verbatim")
    def compile(infile, outfile):
        pass
    

    Results in the following function calls:

    compile("1.c", ["1.o", "1.o"], "Compiling 1", "verbatim")
    compile("2.c", ["2.o", "2.o"], "Compiling 2", "verbatim")
    

    Since r”1” is optional for the output parameter, "\1.o" and ".o" are equivalent. However, strings in other parameters which do not contain r”1” will be included verbatim, much like the string "verbatim" in the above example.

  • filter = regex(matching_regex)

    is a python regular expression string, which must be wrapped in a regex indicator object See python regular expression (re) documentation for details of regular expression syntax Each output file name is created using regular expression substitution with output

  • output = output

    Specifies the resulting output file name(s) after string substitution

  • extras = extras

    Any extra parameters are passed verbatim to the task function

    If you are using named parameters, these can be passed as a list, i.e. extras= [...]

    Any extra parameters are consumed by the task function and not forwarded further down the pipeline.

    If regex(matching_regex) or formatter(...)` is used, then substitution is first applied to (even nested) string parameters. Other data types are passed verbatim.

    For example:

    @transform(["a.c", "b.c"], regex(r"(.*).c"), r"\1.o", r"\1")
    def compile(infile, outfile):
        pass
    

    will result in the following function calls:

    compile("a.c", "a.o", "a")
    compile("b.c", "b.o", "b")
    

See here for more advanced uses of transform.