Altair Panopticon Streams Server Installation and Reference Help

Adding a Python Transform Operator

A Python script can be executed as a data transformation step in the data pipeline.

Steps:

1. On the Application page, click and select Python Transform in the Add Operator pane.

The Python Transform node icon displays in the Graph pane, as well as the properties to be defined in the Operator Settings pane, and the preview of the data in the Schema pane.

The right (outbound) edge allows you to connect to the other operators.

2. In the Operator Settings pane, define or select the following required properties:

Property	Description
Node ID	The ID of the Python Transform operator.
Inputs	The stream of records or input you will be subscribed to.
Interval	The interval of which the data should be published to the output stream (in milliseconds).
Keep Records	Check to retain or not remove flushed elements. This means the entire set of records will be flushed at each interval.
Host	Host of the Python Pyro instance.
Port	Port of the Python Pyro instance.
HMAC Key	The HMAC key that will be used to connect to the Python Pyro instance.
Data Object Name	The data structure (array of dictionaries) that Panopticon will produce, and then will be utilized by the Python Script.
Serialization Type	The serialization type: Serpent or Pickle · simple serialization library based on ast.literal_eval · faster serialization but less secure

NOTE

The Host, Port, HMAC Key, and Serialization Type fields will be hidden if their corresponding properties are set in the Streams.properties file.

Role	Corresponding Property in Streams.properties
Host	connector.python.host
Port	connector.python.port
HMAC Key	connector.python.password
Serialization Type	connector.python.serializertype

3. Enter the required Python Script to execute on the active Pyro instance.

4. Select the Use Apache Arrow check box to enable fast serialization of data frames in the Python transform.

5. In the Input Schema/Sample Data section, the column names of the Input data source are displayed. In cases where there are no rows from the input data source and the Python script is not handling zero rows, you can add sample data to ensure transform is applied.

To add or manage the sample data, you can use the following icons:

Button	Description
	Add sample data for the input column names.
	Select the check box of a sample data row and click to delete, or select the topmost check box and click to delete all of the sample data rows.

6. In the Generate Output Schema section, click Generate Output Schema to fetch the schema of the output topic. This populates the list of columns, with the data type found from inspecting the first ‘n’ rows of the file.

7. Select the Priority of the node's startup:

Priority	Description
APPLICATION	Running and successful completion of the node is critical in the application startup.
HIGHEST	Highest priority but not critical.
HIGH (Default)	High priority but not critical.
STANDARD	Standard priority.
LOW	Low priority.

8. You can also click the following icons:

Button	Description
	Fetch the schema of the output topic. This populates the list of columns, with the data type found from inspecting the first ‘n’ rows of the file.
	Add a new field entry.
	Select the check box of a field entry and click to delete.

9. Save the changes.

Adding a Python Transform Operator

Example