Create a Complex Flow

Everything done so far could have been done just as easily using make or a C-shell script. In this step you will build a flow which neither make nor a shell script could handle efficiently. This is a flow that spans multiple directories.

This example project will dynamically create a set of subdirectories. For each subdirectory, it will run four jobs within that context that process the same input file aa that comes from the top level main directory. The jobs are the ones we have been using to emulate useful work.

This flow definition implements a project that has variant ways of processing, starting from a given input file, with each variant task branch running in a different area, and each one enabled to be run in parallel or in any order, independent of work done in another subdirectory. For this tutorial flow, all the variant task branches have the same set of four jobs which are emulated by the same simple cp commands. In a real project, each variant task sequence could involve different programs. 
% cat Flow2.tcl

for {set i 1 }  { $i < 20 } { incr i } {
    indir -create subdir$i {
        J vw cp ../aa bb
        J vw cp bb cc
        J vw cp cc dd1
        J vw cp cc dd2
    }
}

The for construct is standard Tcl, while the procedure indir is a FlowTracer extension. In this case you want to create the subdirectories subdirN, so you will use the option -create of indir.

  1. Start the sequence with removal of files that might exist if the steps are done again. 
    % vovforget -elements System:nodes
% rm -rf aa subdir*
% vovbuild -f Flow2.tcl 
......................................
......................................
......................................
......................................
% touch aa
% vsr -all


    Figure 1.

    You have to use option -all of vsr because this flow spans multiple directories and the default target of vsr is just the current working directory.

  2. As was done earlier, create a batch shell script to build the flow and run it. 
    % cat dowork2.sh 

vovforget -elements System:nodes
rm -rf aa subdirs*
sleep 5
vovbuild -f Flow2.tcl
sleep 5
touch aa
sleep 5
vsr -all

% sh dowork2.sh
  3. Rather than use a shell script to redo the commands to build and run the flow, you can work with the flow as it is defined in FlowTracer, and try touching aa to emulate a change in the primary input file. This models an event that precedes running all the dependent jobs. Run vsr -all to have FlowTracer schedule and dispatch all the dependent variant processes in the task paths. Watch the state of the nodes as the activity progresses.


    Figure 2.