Makefile Conversion

If you have Makefiles, you can use vovmake to extract flow information from the Makefile. This approach is intrinsically less capable and less reliable than using FlowTracer's Flow Description Language (FDL) to describe your flows, because of the limitations of the Makefiles. For example, the Makefiles do not have information about the resources (e.g. licenses, RAM) required for each job in the flow, in which case you may want to augment the description contained in the Makefile with a vovmake.config.tcl file.

There are some limitations to using Makefiles to maintain large flows:

make is commonly used recursively, so there is no one instance of make that contains the complete dependency graph
make does not have any graphical user interface, so you need to read log and dump files carefully when troubleshooting.
the language of makefiles is complicated and arcane, and uses many tersely-named special variables. This is OK for experts, but is a barrier to new or casual users.

With some exceptions, running your Makefiles with vovmake can provide a valuable visualization of the structure of the Makefiles.

You can use your Makefiles as a starting point, then run your computation with FlowTracer, or continue to maintain the makefiles, and just use FlowTracer to visualize what they are doing.

Often the flow can be brought up-to-date more quickly by FlowTracer than by make, by running multiple jobs in parallel over the network. Since FlowTracer is client-server, all users see the same state of the computation in their GUI or browser.

Prerequisites for Running vovmake

You must first start a FlowTracer project, managed by the vovserver program. This can run on the same or a different host from where you run vovmake.

The command for starting a project is:

% vovproject create <project_name>
% vovproject enable <project_name>

In the simplest form, use vovmake just like you would use your regular make, that is, instead of typing:

% make all

you will type:

% vovmake all

Other common invocations are:

% vovmake -help
% vovmake -clean

vovmake -help

DESCRIPTION:
   This is a script used to convert makefiles into FlowTracer flows.
   1. First the makefiles are interpreted by a modified gmake,
      called gmake_with_vov_extension, which contains some
      Altair Engineering extension and supports the option -F to dump
      the dependencies into Tcl file.
   2. The resulting Tcl file is then mapped into a flow by means of the
      script $(VOVDIR)/tcl/vtcl/vovmaketoflow.tcl.


USAGE:
    % vovmake [VOVMAKEOPTIONS] [gmake_options]

VOVMAKEOPTIONS:
   -help                     -- Print this help.
   -clean                    -- Just cleanup the generated files.
   -nocleanup                -- Do not cleanup temporary files
                                (for debugging).
   -gmake <GMAKEBIN>         -- Use specified gmake binary
                                This must be one with the Altair Engineering
                                extensions.
                                Default gmake_with_vov_extension
   -build "options"          -- Options passed to vovbuild, with flow
                                $(VOVDIR)/tcl/vtcl/vovmaketoflow.tcl.
   -version                  -- Show gmake version and exit
   -wrapper <dfltwrapper>    -- Specify default wrapper for jobs in
                                flow.  It is passed to vovbuild.
   -run [ft|no]              -- Run Flowtracer to get more information about
                                input makefile. Default is ft.

EXAMPLES:
  % vovmake -help

  % vovmake install
  % vovmake <SOME_MAKE_TARGET>
  % vovmake -clean

  % vovmake -nocleanup  all
  % vovmake -gmake  ~/bin/my_gmake_with_vov_extension
  % vovmake -wrapper vrt
  % vovmake -build "-env BASE" install


To find the options of vovmaketoflow.tcl, you can use
  % vovbuild -f  $VOVDIR/tcl/vtcl/vovmaketoflow.tcl -- -help

How vovmake Works

The utility vovmake operates as follows:

It calls gmake_with_vov_extension to parse the Makefiles and generates an intermediate flow file called /tmp/vovmakeUSERNAME/TIMESTAMP/vovmakePID.tcl.
This intermediate flow file is then processed by the Tcl script vovmaketoflow.tcl which transfers the dependencies derived from the Makefiles into the FlowTracer server.
The processing of the Makefile rules occurs as follows:
- Simple rules (one-line commands) are preserved and appear as a job in the flow
- Complex rules, i.e. rules that have either multiple lines, or multiple commands, or redirections, are encapsulated into a local shell script in the directory vmake_scripts
- The dependencies in your flow graph will be those in your makefile
vovmake issues the command vsr to request that the flow graph should be brought up-to-date

Limitations on makefiles

There are many different versions of make available, that support different commands. GNU make is the most common, and it has some functionality that is not supported by vovmake.

Makefiles that simply define variables, targets, and recipes to make the targets should work OK.

Suggestions

Use vovmake only to "build" stuff, not to run targets that clean up. Many makefiles have a target named 'clean' or the like which will remove intermediate and output files, leaving only the sources. It is common to compensate for incorrect or incomplete dependencies by using the 'make clean; make all' sequence.

When dealing with recursive make, we recommend you start from the leaves, i.e. the deepest directories. Try to do one directory at a time.

Some Results Using Compilation Examples

The following table describes the results obtained at Altair by using vovmake on some common open-source packages.


Package Name	Comments	Size of resulting Flow
GNU make 3.80	One shot build with the usual: `% ./configure % vovmake`	less than 400 nodes
mysql 5.0.19	Complete build succeeds. You can observe multiple passes: `% ./configure % vovmake`	1000 jobs and 4400 files
GNU gdb 6.3	On Linux, you get a flow graph with about 2300 nodes. Exclude the "maybe-*" targets by adding `vtk_exclude_rule -regexp {/maybe-}` to exclude.tcl file. `% ./configure % vovmake % vovmake % vsr -f -all -retry 4`	About 2300 nodes
valgrind 2.4.0	The package almost builds valgrind in one shot, but shows a very "interesting" makefile system with a few conflicts between addrcheck and memcheck and some failures because of missing '*.Tpo' dependencies. `% ./configure % vovmake % vovmake % vsr -f -all -retry 4`	About 120