Connector for Google Cloud Pub/Sub
The Google Cloud Pub/Sub connector allows connection to Google Cloud Pub/Sub’s message bus on a real-time streaming basis. Specifically, the connector allows Panopticon to subscribe to XML, JSON, TEXT or FIX based messages that are published on particular topics. The data format itself is arbitrary, and consequently, the connection includes the message definition.
Steps:
1. Enter the Service Account Credential JSON Text with the generated JSON key (contains the private key) in the following format:
NOTE |
Ensure that when parameterizing the values in the Credential JSON Text, there is no white space as a single line content. |
2. Click to populate the Topic
drop-down list. Initially, the first topic in the list is displayed
in the Topic drop-down box.
Select a topic.
3. Click to populate the Subscription
Name drop-down list and select a subscription name.
You can also opt to create a subscription by manually entering the value into the Subscription Name list box.
NOTE |
· A subscription name will be automatically generated when it is not entered or selected in the drop-down list. This subscription will be created for connection and will be deleted as soon as its work is done. For example, when starting a presentation mode, a subscription will be created. Upon quitting the presentation mode, the subscription will then be deleted. · Pub/Sub can automatically delete inactive subscriptions. This can be done by configuring the minimum required time of inactivity to schedule a subscription for deletion. This time must be longer than the message retention duration. |
4. Select the Message Type.
5. Select either the dot (.) or comma (,) as the Decimal Separator.
6. Click to the fetch the schema based
on the connection details. Consequently, the list of columns with
the data type found from inspecting the first ‘n’ rows of the input
data source is populated and the Save button is enabled.
This also populates the Id Column with the set of columns, of arbitrary type, that can be concatenated to form a unique row identifier.
7. You can also opt to load or save a copy of the column definition.
8. Click to
add columns to the
Google Cloud PubSub connection that will represent sections of the
message.
9. Provide the following information:
Property |
Description |
Name |
The column name of the source schema. |
Fix Tag/XPath/Column Index/Json Path |
The Fix Tag/XPath/Column Index/Json Path of the source schema. |
Type |
The data type of the column. Can be a Text, Numeric, or Time |
Date Format |
The format when the data type is Time. |
Enabled |
Determines whether the message field should be processed. |
To
delete a column, check its or all the column entries, check
the topmost
,
then click
.
10. Google Cloud Pub/Sub messages can have additional metadata as custom attributes.
Panopticon Google Cloud Pub/Sub connector supports reading these attributes as column values. The generate column logic automatically checks and generates attribute columns if messages received contain attributes.
Additionally, like columns from message data, you can manually add them by clicking . A new entry displays.
Name can be any unique column name within the data source. The attribute name must match to an attribute name in message otherwise it will be treated as null value. Currently all attribute columns are treated as Text columns, we can't change column type.
Check the Enabled box to enable an attribute column.
To delete a column,
check its or
all the column entries, check the topmost
, then click
.
11. Date/Time values of output data and Date/Time inputs, where supported, is by default unchanged.
You can opt to define the Show in Timezone and Source Timezone settings.
12. For this section:
Refer to Define Real-Time Settings for more information.