Defining Real-time Settings

Streaming connectors have a common section for defining real-time settings. Follow the steps below to select a key column or concatenated key for the streaming time series window.

Steps:

1.  After generating columns or fetching schema on the streaming connector, the list of columns with the data type found from inspecting the first ‘n’ rows of the input data source is populated.

Consequently, on the Real-Time Settings section, the Id Column drop-down list displays with the set of columns, of arbitrary type.

2.  Select a key column to manage data updates and inserts. In some cases, select multiple key columns to form a unique row identifier.

Furthermore, a streaming time series window can be generated by creating a compound key with the Id Column, plus a separately specified Time ID column.  This Time ID column can be from the source dataset, or alternatively automatically generated.

If the Time Id column is selected, then a scrolling time window can be specified.

For Automatic Time Id, define the Time Id Column Name.

As new data arrives from the subscription, new time slices will automatically be added, and old ones will be deleted.

If a new ID is received, a new row is added to the in-memory data set representing the connector's  topic subscription. While if an existing ID is received, an existing row is updated.

3.   Check the Reset Data on Reconnect box to flush out the stale data and reload data after reconnection.