Automatic Zipping and Unzipping Files

Rule 1: Compression

A zippable file is automatically compressed when all the jobs that require compression have completed successfully. The check to see if compression is required is performed about every 30 seconds.

Rule 2: Decompression

A zippable file is automatically uncompressed if any of the jobs that requires it is queued and ready to be executed. Jobs will not be executed with compressed inputs.

If a file is compressable, it can be in either the normal or the compressed state. For a file called 'X', if the file does not exist but 'X.gz' does, then the file is considered compressed.

Directing vovzip Jobs to Certain Hosts

The work to compress/uncompress a file is performed by a system job that is executed by the owner of the project. These jobs are scheduled to be executed by the appropriate vovtaskers.

To direct the vovzip jobs to particular hosts, create a tool map in the project's resources.tcl file, where the right-hand side of the map is the necessary resource.

vtk_resourcemap_set Tool:vovzip UNLIMITED vovzip_host

The vovtasker resource vovzip_host can be placed on the desired hosts in the project's taskers.tcl file. The number of concurrent vovzip jobs can be limited: instead of UNLIMITED, insert an integer such as 10.

Defining Zippable Files

J vw cp aa bb
J vw cp bb cc
Z bb cc

Another method is using the command vovset with the subcommand zippable.

% vovset zippable SETNAME 1
% vovset zippable SETNAME 0

set id [ vtk_place_find "FILE" "aa"]
vtk_place_get $id info
puts "The current value of the zippable flag is $info(zippable)"
puts "The current value of the zipped   flag is $info(zipped)"

set  info(zippable)  1
vtk_place_set $id info

Recommendations for Zippable Flags

Using the zippable flag is recommended for relatively large files, such as greater than 100kB. A tradeoff needs to be evaluated: the space consumed and flow complexity and additional CPU/IO time, versus the degree of the compressibility of the files. For example, JPEG and PDF files are already highly compressed; zipping these files may produce little or not benefit. Test vector files, however often compresses to 3-8% of their native size.

vovzip Jobs and Log Files