Configure Container Integration

Linux containers can be leveraged to constrain the amount of system resources used by jobs. Accelerator's container support is designed to be agnostic of the container solution. The examples provided are for Docker specifically.

File: Container C1 definition:

### Example named-container configuration c1.
## Hook run directory
#
# The default behavior is to switch to the job directory before running the
# hooks. If this directory is not accessible until the hooks have been executed
# though, the job will fail. This issue can be resolved by setting
# containerHooksRunDir to a location that is guaranteed to be accessible prior
# to the hooks being executed. This causes the subtasker to switch to that
# directory instead, and the original job directory will be made available to
# the hooks via the VOV_CONTAINER_JOB_RUNDIR environment variable. The value of
# this variable can then be used by the hooks to specify the working directory
# for the container instance.
# containerHooksRunDir "/path/to/run/directory"

## Hook definitions
#
# containerHook <type> <mode> <path> <signature>
#   <type>      = setup | enter | cleanup | teardown
#   <privilege> = user  | root
#   <file>      = absolute path to hook file
#   <signature> = output of vovsignfile command
#
# Specifies which hook(s) should be called to interact with the container
# platform throughout the job's life cycle:
#
# 1. The setup hook is required if the container must be created prior to
#    spawning the job. This hook will always be used if root privileges are
#    required to create containers.
# 2. The enter hook is always required and is responsible for placing the
#    job into the container. If root privileges are not required to create
#    containers and the container platform supports it, the enter hook can
#    also create the container, as would be done via the "docker run" command,
#    for example. The enter hook must block for the duration of the job.
# 3. The cleanup hook can be used to remove temporary artifacts that are
#    generated by the job.
# 4. The teardown hook is required to stop and/or remove the container. This
#    hook would normally be used if a setup hook is used to create the
#    container.
#
# Any root-mode hook must be owned by root on the filesystem.
#
# The tasker sets at least the HOST, VOV_PROJECT_NAME, VOV_JOBSLOT, and
# VOV_CONTAINER_NAME environment variables in the job execution environment.
# It is intended for the hooks to use these variables to uniquify container
# instances. Hooks can be tested by running them manually in the shell as long
# as the required environment variables are set to some value. The only hook
# that is called with arguments is the enter hook, which expects the job command
# line to be passed in. An example test run of the enter hook may look like:
#
# % env HOST=foo VOV_PROJECT_NAME=vnc VOV_JOBSLOT=0 VOV_CONTAINER_NAME=c1 \
# /path/to/c1-enter.sh whoami
# containerHook setup    root "/path/to/c1-setup.sh"    1211238920
containerHook enter    user "/path/to/c1-enter.sh"    97261574
# containerHook cleanup  user "/path/to/c1-cleanup.sh"  5376821904
# containerHook teardown root "/path/to/c1-teardown.sh" 8237156649

It is recommended to manually test each hook before using them in production environment. Use the tasker hosts for testing and production. An example test run of the "enter" hook may look like:

#% env HOST=foo VOV_PROJECT_NAME=vnc VOV_JOBSLOT=0 VOV_CONTAINER_NAME=c1 \
#/path/to/c1-enter.sh whoami

File: c1-enter.sh

!/bin/bash -fxv
# Note: The enter hook must block for the duration of the job.
#
# Container c1 example enter hook: an all-in-one script that creates a
# container, launches a job inside of it, then exits and removes the container.
#
 
 
# The following environment variables are available and should be used to avoid
# container and/or host name conflicts:
#
#   HOST                (string)
#   VOV_PROJECT_NAME    (string)
#   VOV_JOBSLOT         (number)
#   VOV_CONTAINER_NAME  (string)
#
# The following environment variables are available if defined in the named
# container configuration file:
#
#   VOV_CONTAINER_CORES (number)
#   VOV_CONTAINER_RAM   (megabytes)
#   VOV_CONTAINER_TMP   (megabytes)

# For containers handling interactive jobs (-I), uncomment the following line.
# set -m
containerName=${VOV_PROJECT_NAME}_${HOST}_${VOV_CONTAINER_NAME}_${VOV_JOBSLOT}
uid=$(id -u ${USER})
# Handle the run directory specified in the named container configuration file.
if [[ -d $VOV_CONTAINER_JOB_RUNDIR ]]; then
  workDirOptions="--workdir $VOV_CONTAINER_JOB_RUNDIR --mount type=bind,source=${VOV_CONTAINER_JOB_RUNDIR},target=${VOV_CONTAINER_JOB_RUNDIR}"
else
  workDirOptions="--workdir $PWD"
fi
# Process limits into Docker options.
limitOptions=""
if [[ -n $VOV_CONTAINER_CORES && $VOV_CONTAINER_CORES > 0 ]]; then
  limitOptions+=" --cpus $VOV_CONTAINER_CORES"
fi
if [[ -n $VOV_CONTAINER_RAM && $VOV_CONTAINER_RAM > 0 ]]; then
  ramSpec="${VOV_CONTAINER_RAM}m"
  limitOptions+=" --memory $ramSpec"
fi
if [[ -n $VOV_CONTAINER_TMP && $VOV_CONTAINER_TMP > 0 ]]; then
  # Use in-memory tmpfs for /tmp in container.
  tmpBytes=$(($VOV_CONTAINER_TMP*1048576))
  limitOptions+=" --mount type=tmpfs,destination=/tmp,tmpfs-size=$tmpBytes"
fi
# Capture job environment in a file for Docker to import.
envFile=/tmp/${containerName}.env
env > $envFile

# Use the Docker "run" command to create a container based on the "myImage"
# container image, setup networking, specify the user, capture the environment,
# and bind-mount the required directories for the job. Finally, the job itself
# is passed in for Docker to execute. The image must have the ability to resolve
# the job owner's UID, have access to the VOV software installation, and be able
# to execute the vw job wrapper along with the job command.
# The "$@" variable will contain "vw <jobCmd> <jobArgs>".

The configuration for each hook that is defined must contain a signature. The signature is used by the tasker to verify the hook has not been tampered with since it was defined by the administrator. The signature can be obtained by using the vovsignfile utility. In the following example, the signature is highlighted in red:

# containerHook setup    root "/path/to/c1-setup.sh"    1211238920
containerHook enter    user "/path/to/c1-enter.sh"    97261574
# containerHook cleanup  user "/path/to/c1-cleanup.sh"  5376821904
# containerHook teardown root "/path/to/c1-teardown.sh" 8237156649