SDS Configuration

On startup, the vovserver will create and/or update the following configuration items:

SDS configuration directory, at the server working directory (SWD)/config/publishers/sds, for example, ../vnc.swd/config/publishers/sds
SDS configuration file in the SDS configuration directory, sds.cfg
Avro schema files (updated each time the server starts)

If the sds.cfg file does not already exist, the following default sds.cfg file is created:

{
  enabled = 0,
  kafka_servers = "",
  format = "json",
  site = "",
  group = "",
  enable_jobdata = 0,
  events = {
    project = {
      schemaId = 0,
      topic = "vov-projects",
    },
    taskers = {
      schemaId = 0,
      topic = "vov-metrics-taskers",
    },
    jobs = {
      schemaId = 0,
      topic = "vov-metrics-jobs",
    },
    scheduler = {
      schemaId = 0,
      topic = "vov-metrics-scheduler",
    },
    tasker = {
      schemaId = 0,
      topic = "vov-metrics-tasker",
    },
    taskerlist = {
       schemaId = 0,
       topic = "vov-metrics-taskerlist",
    },
    jobdata = {
       schemaId = 0,
       topic = "vov-jobdata",
   },
   deletejob = {
      schemaId = 0,
      topic = "vov-deletejob",
   },
  },
}

Configuration File Parameters

Service level configurable parameters in sds.cfg:


Parameter	Values	Default	Description
enabled	0 or 1	0	Disable/enable SDS on startup/readconfig
format	"json" "confluent" azure"	"json"	Specifies the Kafka payload format to use: "json": use plain text using the JSON format "confluent": use AVRO encoding with the schema registry identifier used in the Confluent services "azure": use the AVRO encoding with the schema registry identify used in Azure's EventHub service.
site	string	""	User definable string to be delivered with the project record
group	string	""	User definable string to be delivered with the project record
debug	0 or 1	0	Disable/enable SDS debug logging on startup/readconfig
enable_jobdata	0 or 1	0	Disable/enable job events
enable_jobarraydata	0 or 1	Enabled	Disable/enable job array events. There will be events for each job in the array.
enable_systemjobs	0 or 1	Disabled	Disable/enable system job events
enable_resyncjobdata	0 or 1	Disabled	Enable or disable the RESYNC operation for job data.
jobdata_resync_period_hours	integer	24 hours	The time interval (in hours) at which the resync operation will be triggered.
jobdata_resync_window_hours	integer	4 hours	The window of time (in hours) during which the resync operation should be completed.
jobdata_resync_chunk	integer	1000	The maximum number of resync events sent at a time.

Event Specific Configurable Parameters in sds.cfg

Each event has its own configuration section in the config file, for example, for the project event:

  ...
  events = {
    project = {
      schemaId = 0,
      topic = "vov-projects",
    },
  ...


Parameter	Values	Default	Description
schemaId	integer or string	0	N is a string, publish using Azure EventHub encoding where N is the schema's registry ID N < 0, do not publish this event N = 0, publish using single object encoding N > 0, publish using Confluent encoding where N is the schema's registry ID
topic	string	see event table	The name of the Kafka topic to which these events are published.
format	"" "json" "confluent" "azure"	""	Overrides format for this event "" = use format specified at the overall service level

Kafka Configuration

The Kafka libraries enable some flexibility with configuration. For example, in the publishers/sds/sds.cfg file:

  # kafka properties forwarded to rdkafka library for client config
  kafka_properties = {
    security = {
      protocol = "ssl",  # security.protocol=ssl
    },
    ssl = {
      ca = {
        location = "/home/rhenry/Proj/confluent/ssl2/ca-cert" # ssl.ca.location=/home/rhenry/Proj/confluent/ssl2/ca-cert
      },
      certificate = {
        location = "/home/rhenry/Proj/confluent/ssl2/client_hecto_client.pem" # ssl.certificate.location 
      },
      key = {
        location = "/home/rhenry/Proj/confluent/ssl2/client_hecto_client.key", # ssl.key.location
        password = "abcdefgh" # ssl.key.password
      }
    }
  }

Kafka uses security.protocol = "ssl"

However, there is flexibility to include an equivalent as security { protocol = "ssl" }.

The vovserver/SDS is passing the configuration through to the underlying Kafka libraries and network layer, making this feasible.

Change the Config File for the First Time

In order to use SDS for the first time, the user must perform the following operations:

Set the kafka_servers parameter in the sds.cfg file to the bootstrap server(s) for their kafka installation; for example, kafka_servers = "kafkahost:9092" or kafka_servers = "kafkahost1:9092,kafkahost2:9092"
If publishing using the Confluent Schema Registry, then the following steps are also needed:
1. Upload the schema files to the schema registry and note the IDs assigned to each schema.
2. Assign the schema registry IDs discovered in step 1 to the events in the sds.cfg file (see Event Specific configuration below)

For the initial release the Kafka published events are:


Event Name	Description	Schema File	Default Topic
project	(relatively) Static project information that may be useful to join with time series data	vov.projects.avsc	vov-projects
taskers	Metrics related to the state and capacity of the taskers	metrics.taskers.avsc	vov-metrics-taskers
jobs	Metrics related to the number of jobs in specific states and rate of dispatch/completion	metrics.jobs.avsc	vov-metrics-jobs
scheduler	Metrics related to scheduler performance, sizes, clients, innerloop timers	metrics.scheduler.avsc	vov-metrics-scheduler

The following events have been added in the 2022.1.0 release:


Event Name	Event Type	Description	Schema File	Default Topic
deletejob	DELETEJOB	Information about deleted job like jobid, user, name, host, etc/	vov-deletejob.avsc	vov-deletejob
jobdata	INITIAL	Initial job data like jobid, command, jobclass, status, etc.	vov-jobdata.avsc	vov-jobdata
jobdata	UPDATE	Updated job data like jobid, command, jobclass, status, etc.	vov-jobdata.avsc	vov-jobdata
jobdata	PROPERTYADD	Information about job new properties in the format <name value>	vov-jobdata.avsc	vov-jobdata
jobdata	PROPERTYMODIFY	Information about job modified properties in the format <name value>	vov-jobdata.avsc	vov-jobdata
jobdata	PROPERTYDELETE	Information about job deleted properties in the format <name value>	vov-jobdata.avsc	vov-jobdata

Changing the Config File at Run Time

The SDS configuration may be changed while the server is running.

The SDS service may be enabled/disabled by using the command:
```
 $ vovservermgr config sds.enabled 1/0 
```
Update the config file for the running server and/or publish a new project event with the following command:
```
$ vovservermgr config sds.readconfig 1
```

The debug setting may be enabled using:

$ vovservermgr config set_debug_flag SDS
$ vovservermgr config reset_debug_flag SDS

Troubleshooting

If the kafka_servers cfg parameter is not set correctly, the server log will contain entries like the following:

%3|1610382360.585|FAIL|rdkafka#producer-1| [thrd:foo:9092/bootstrap]: foo:9092/bootstrap: Failed to resolve 'foo:9092': Temporary failure in name resolution (after 1033ms in state CONNECT)
%3|1610382360.585|ERROR|rdkafka#producer-1| [thrd:foo:9092/bootstrap]: 1/1 brokers are down
%3|1610382363.544|FAIL|rdkafka#producer-1| [thrd:foo:9092/bootstrap]: foo:9092/bootstrap: Failed to resolve 'foo:9092': Temporary failure in name resolution (after 993ms in state CONNECT, 1 identical error(s) suppressed)

If the kafka servers are not running or reachable, the server log will contain entries like the following:

%3|1610383215.659|FAIL|rdkafka#producer-2| [thrd:hecto:9092/bootstrap]: hecto:9092/bootstrap: Connect to ipv4#127.0.1.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)