See also

• @collate in the Ruffus Manual
• Use of add_inputs(...) | inputs(...) in the Ruffus Manual
• Decorators for more decorators

@collate( input, filter, replace_inputs | add_inputs, output, [extras,...] )

Purpose:

Use filter to identify common sets of inputs which are to be grouped or collated together:

Each set of inputs which generate identical output and extras using the formatter or regex (regular expression) filters are collated into one job.

This variant of @collate allows additional inputs or dependencies to be added dynamically to the task, with optional string substitution.

add_inputs nests the the original input parameters in a list before adding additional dependencies.

inputs replaces the original input parameters wholescale.

This is a many to fewer operation.

Only out of date jobs (comparing input and output files) will be re-run.

Example of add_inputs

regex(r".*(\..+)"), "\1.summary" creates a separate summary file for each suffix. But we also add date of birth data for each species:

animal_files = "tuna.fish", "shark.fish", "dog.mammals", "cat.mammals"
# summarise by file suffix:
@collate(animal_files, regex(r".+\.(.+)$"),  add_inputs(r"\1.date_of_birth"), r'\1.summary')
def summarize(infiles, summary_file):
    pass

This results in the following equivalent function calls:

summarize([ ["shark.fish",  "fish.date_of_birth"   ],
            ["tuna.fish",   "fish.date_of_birth"   ] ], "fish.summary")
summarize([ ["cat.mammals", "mammals.date_of_birth"],
            ["dog.mammals", "mammals.date_of_birth"] ], "mammals.summary")

Example of add_inputs

using inputs(...) will summarise only the dates of births for each species group:

animal_files = "tuna.fish", "shark.fish", "dog.mammals", "cat.mammals"
# summarise by file suffix:
@collate(animal_files, regex(r".+\.(.+)$"),  inputs(r"\1.date_of_birth"), r'\1.summary')
def summarize(infiles, summary_file):
    pass

This results in the following equivalent function calls:

summarize(["fish.date_of_birth"   ], "fish.summary")
summarize(["mammals.date_of_birth"], "mammals.summary")

Parameters:

• input = tasks_or_file_names

  can be a:

  1. Task / list of tasks.

     File names are taken from the output of the specified task(s)

  2. (Nested) list of file name strings (as in the example above).

     File names containing *[]? will be expanded as a glob.

     E.g.:"a.*" => "a.1", "a.2"

• filter = matching_regex

  is a python regular expression string, which must be wrapped in a regex indicator object See python regular expression (re) documentation for details of regular expression syntax

• filter = matching_formatter

  a formatter indicator object containing optionally a python regular expression (re).

• add_inputs = add_inputs(...) or replace_inputs = inputs(...)

  Specifies the resulting input(s) to each job.

  Positional parameters must be disambiguated by wrapping the values in inputs(...) or an add_inputs(...).

  Named parameters can be passed the values directly.

  Takes:

  1. Task / list of tasks.

     File names are taken from the output of the specified task(s)

  2. (Nested) list of file name strings.

     Strings will be subject to substitution. File names containing *[]? will be expanded as a glob. E.g. "a.*" => "a.1", "a.2"

• output = output

  Specifies the resulting output file name(s).

• extras = extras

  Any extra parameters are passed verbatim to the task function

  If you are using named parameters, these can be passed as a list, i.e. extras= [...]

See @collate for more straightforward ways to use collate.