[13] Example Applications

AggregationExample

Demonstrates how to aggregate data based on a grouping key and a set of aggregated fields.

Includes simple aggregations such as avg, count, first, last, max, min, samples, sum, sdevp, sdevs, Sum, varp, and vars.

BranchExample

Demonstrates how to split a stream into one or more branches.

CalculateRemoveReplaceNull

Demonstrates how to:

• Remove and replace fields from output schemas

• Set a field value to null

• Set a field value to the current timestamp

CalculationExample

Includes the SquareRoot calculation.

CalculationsExample

Includes the following calculations:

• Numeric calculations such as Abs, SquareRoot, Subtract, Multiply, Divide, Truncate, IF

• Text calculations such as Upper, Lower, Proper, Left, Right, Mid, Concat, Find

• Time Period calculations such as DateDiff

In addition, data type casting between Text, Number, and Date/Time

ConflateExample

Demonstrates how to lower the frequency of updates by setting a fixed interval.

EmailExample

Shows how to send an email via SMTP where the SMPT and email settings can be parameterized. Each record passed to the connector results in an email which can be primarily used as an output for alerting, having a conditional expression that would need to be fulfilled for a record to be forwarded to the output.

ExternalInputExample

Demonstrates how to directly source data from a Kafka topic (defined in the schema registry with the message format set to Avro).

ExternalInputJsonParserExample

Demonstrates how to directly use a parsed input Json data.

ExternalInputXMLParserExample

Demonstrates how to directly use a parsed input XML data.

FilterExample

Demonstrates how to filter a data source based on a predicate.

InfluxDBExample

Allows periodical dumping of records from a Kafka topic into an InfluxDB 1.x output connector. Requires the InfluxDBWriter plugin.

JDBCExample

Allows periodical dumping of records from a Kafka topic into a JDBC database output connector. Requires the JDBCWriter plugin.

JoinExample

Demonstrates how to join a stream to a global table.

KdbExample

Allows periodical dumping of records from a Kafka topic into a Kx kdb+ output connector. Requires the KdbWriter plugin.

MetronomeExample

Demonstrates how the metronome operator works in generating a timestamp field schema. A static metronome has a defined frequency while a dynamic  metronome takes frequency as an input which determines the speed of the simulation.

RetentionTimeExample

Demonstrates how to define the different retention time periods set for tables, input streams, output streams, and topics in an application.

This helps minimize memory utilization and the amount of data retrieved when subscribing from the beginning to the latest messages.

NOTE: Setting these properties in the application level overrides the defaults set in the Streams.properties file.

For example, if the following properties are defined in the Streams.properties file:

In the application level, the input retention period will be 1,000 milliseconds instead of 60,000 and the output retention period will be 1,000 milliseconds instead of 900,000. Also, a custom topic retention period has been added using the following pattern: TopicName.retention.ms (i.e., TimeSeries.retention.ms).

StockMarketSimulator

Shows a stock market simulation using a streaming data with join, calculations, and metronome operators.

StockStaticTimeSeriesApp

Joins a static and a time series data sources using common keys. Also demonstrates adding a sum aggregation.

StreamtoGlobalTableJoinExample

Joins stream and global table inputs using common keys.

StreamToTableJoinExample

Joins stream and table inputs using common keys.

TextExample

Allows periodical dumping of records from a stream Kafka topic into a Text connector. Requires the TextWriter plugin.

UnionExample

Unioning of two streams.

WindowedStreamExample

Demonstrates aggregation across a windowed stream.